Image processing device, imaging device, image processing method, and program

ABSTRACT

The number of light sources of original image data and the types of light source are determined (S11). A reference white balance gain which is set for each light source type of the original image data is acquired (S12). An influence rate of each light source type is acquired for each pixel of the original image data and a mixture ratio of the reference white balance gains is acquired on the basis of the influence rate (S13). The reference white balance gains and the mixture ratio are stored in a storage medium so as to be associated with the original image data (S14). It is possible to acquire the white balance gain for each pixel required for multi-area white balance processing for the original image data from the reference white balance gains and the mixture ratio stored in the storage medium subsequently.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Divisional of copending U.S. patent applicationSer. No. 15/276,251 filed Sep. 26, 2016, which is a Continuation of PCTInternational Application No. PCT/JP2015/057276 filed on Mar. 12, 2015,which claims priority under 35 U.S.C. § 119(a) to Japanese PatentApplication No. 2014-71682 filed on Mar. 31, 2014. Each of the aboveapplications is hereby expressly incorporated by reference, in theirentirety, into the present application.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an image processing device, an imagingdevice, an image processing method, and a program, and moreparticularly, to a multi-area white balance processing technique.

2. Description of the Related Art

Multi-area white balance processing (hereinafter, referred to as“multi-area WB processing”) is white balance processing (hereinafter,referred to as “WB processing”) that applies different white balancegains (hereinafter, referred to as “WB gains”) to each pixel or eacharea of one image. According to multi-area WB processing, even in animage of an object in which light components are emitted from aplurality of light sources to each pixel (each area) at differentratios, it is possible to accurately correct the color (color balance)of each pixel (each area).

For example, in a case in which flash light is emitted to capture anight portrait scene, an image of a person is captured while beingaffected by flash light (for example, light including a large amount ofblue-wavelength light) and the background (night scene) of the capturedimage is not substantially affected by the flash light and is affectedby other types of environmental light such as light from a sodium lamp(light including a large amount of red-wavelength light). In this case,it is necessary to apply a WB gain for cancelling the influence of flashlight to the image of a person in order to obtain a good person imagecolor (color balance) using WB processing. It is necessary to apply a WBgain for cancelling the influence of other types of environmental lightto a background image in order to obtain a good background color (colorbalance). Therefore, in a case in which a common WB gain is applied toall of the pixels forming an image, it is difficult to obtain both agood person color (color balance) and a good background color (colorbalance).

However, according to multi-area WB processing, a WB gain applied to aperson portion and a WB gain applied to a background portion in oneimage are changed. Therefore, for example, the WB gain applied to eachpixel is optimized according to the amount of flash light that reachesthe pixel to obtain both a good person color (color balance) and a goodbackground color (color balance).

JP2012-165077A discloses a white balance control device that can preventa variation in a white balance control value which is calculated usingcolor information obtained from an object region. The white balancecontrol device detects the object region from an image, divides theimage into a plurality of regions, calculates color informationindicating the representative colors of regions that overlap the objectregion, and determines a weight. Then, the white balance control deviceapplies the weight determined for each region that overlaps the objectregion to each color information item, integrates the color information,and calculates a white balance control value at which the representativecolor becomes a target color from the integration result of the colorinformation and information about a predetermined target color.

JP2010-187113A discloses an imaging device that can acquire an imageadjusted to the color temperature desired by a user. The imaging devicecalculates a WB gain for background light on the basis of a preliminaryimage captured when flash light is not emitted, compares the brightnessof the preliminary image with the brightness of a main image which iscaptured while flash light is emitted immediately after the preliminaryimage is captured, and calculates a WB gain for object light on thebasis of the image data of a region of the main image in which thebrightness difference is greater than a threshold value. Then, theimaging device calculates a plurality of different WB gains on the basisof the WB gain for the background light, the WB gain for the objectlight, and a predetermined WB gain for flash light and individuallyapplies the plurality of WB gains to the main image to generate aplurality of bracket images with different color temperatures.

In some cases, image data and a WB gain are stored in order to adjustthe white balance of an image after the image data is stored.

For example, an imaging device disclosed in JP2009-4895A storesaccessory information required for a development process and RAW imagedata of a main image in a storage medium and can perform auto whitebalance adjustment when the development process is performed for the RAWimage data.

SUMMARY OF THE INVENTION

As described above, since both the RAW image data and the accessoryinformation required for a development process are stored, it ispossible to adjust the white balance of the RAW image data subsequently.

However, in a case in which the white balance of image data, such as RAWimage data, is adjusted subsequently, flexibility in the adjustment ofthe white balance is low.

For example, in some cases, in an image that is captured while flashlight is emitted in a night portrait scene, the user “wants to slightlycorrect only a person portion to the red side”. In this case, when thewhite balance of the entire screen is shifted, the colors (colorbalance) of both the person portion and a background portion are changedand it is difficult to meet the demand of the user.

In a case in which the user manually sets a WB gain for each pixel whenthe white balance is adjusted subsequently, it is possible to meet thedemand of the user who “wants to slightly correct only an image portionto the red side”. However, it takes a lot of time and effort to manuallyset the WB gain for each pixel, which is inefficient.

As such, in a case in which information about the individual WB gainsset for each pixel is stored together with image data, such as RAW imagedata, in order to perform multi-area WB processing, it is possible toswitch between the execution and non-execution of the multi-area WBprocessing subsequently. However, for example, it is difficult to simplymeet the demand of the user who “wants to slightly correct only a personportion to the red side”

The invention has been made in view of the above-mentioned problems andan object of the invention is to provide a technique that can performwhite balance adjustment with high flexibility even in a case in whichmulti-area WB processing is performed subsequently.

An aspect of the invention relates to an image processing devicecomprising: a light source type determination unit that determines thenumber of light sources of original image data and the types of lightsource; a gain acquisition unit that acquires a reference white balancegain which is set for each light source type of the original image data;a mixture ratio acquisition unit that acquires an influence rate of eachlight source type for each pixel of the original image data and acquiresa mixture ratio of the reference white balance gains on the basis of theinfluence rate; and a recording unit that stores the reference whitebalance gains and the mixture ratio in a storage medium so as to beassociated with the original image data.

According to this aspect, the reference white balance gains and themixture ratio are stored so as to be associated with the original imagedata. Therefore, even in a case in which the original image data storedin the storage medium is read and multi-area WB processing is performedsubsequently, it is possible to perform appropriate multi-area WBprocessing by reading the reference white balance gains and the mixtureratio from the storage medium and using the reference white balancegains and the mixture ratio. When the multi-area WB processing isperformed subsequently, it is possible to adjust the reference whitebalance gains and the mixture ratio. Therefore, it is possible toperform white balance adjustment with high flexibility.

Preferably, the light source type determination unit determines that thelight sources of the original image data are flash light andenvironmental light and the gain acquisition unit acquires a referencewhite balance gain for flash light which is set in a case in which thelight source type is the flash light and a reference white balance gainfor environmental light which is set in a case in which the light sourcetype is the environmental light. Preferably, the mixture ratioacquisition unit acquires the influence rate of each of the flash lightand the environmental light for each pixel of the original image dataand acquires the mixture ratio of the reference white balance gain forflash light and the reference white balance gain for environmental lighton the basis of the influence rate. Preferably, the recording unitstores the reference white balance gain for flash light, the referencewhite balance gain for environmental light, and the mixture ratio in thestorage medium so as to be associated with the original image data.

According to this aspect, it is possible to perform appropriatemulti-area WB processing for the original image data captured whileflash light is emitted subsequently. Therefore, it is possible toperform white balance adjustment with high flexibility.

Preferably, the image processing device further comprises a flash imageacquisition unit that acquires flash emission image data which iscaptured while flash light is emitted and flash non-emission image datawhich is captured when flash light is not emitted. Preferably, themixture ratio acquisition unit acquires the influence rate of each ofthe flash light and the environmental light from the flash emissionimage data and the flash non-emission image data and acquires themixture ratio of the reference white balance gain for flash light andthe reference white balance gain for environmental light on the basis ofthe influence rate.

According to this aspect, it is possible to perform multi-area WBprocessing considering the influence rate of the flash light and theenvironmental light subsequently.

Preferably, the mixture ratio acquisition unit acquires a firstbrightness value of each pixel of the flash non-emission image data,acquires a second brightness value of each pixel of the flash emissionimage data, acquires the influence rate of each of the flash light andthe environmental light on the basis of the first brightness value andthe second brightness value, and acquires the mixture ratio of thereference white balance gain for flash light and the reference whitebalance gain for environmental light on the basis of the influence rate.

According to this aspect, it is possible to accurately calculate themixture ratio of the reference white balance gain for flash light andthe reference white balance gain for environmental light on the basis ofthe brightness values (the first brightness value and the secondbrightness value) of the flash emission image data and the flashnon-emission image data.

Preferably, the original image data is the flash emission image data.

According to this aspect, it is possible to perform multi-area WBprocessing for the flash emission image data subsequently.

Preferably, the image processing device further comprises a light sourcetype data acquisition unit that acquires first image data indicating aninfluence of a first light source type and second image data indicatingan influence of a second light source type. Preferably, in a case inwhich the light source types of the original image data determined bythe light source type determination unit include the first light sourcetype and the second light source type, the gain acquisition unitacquires the reference white balance gain set for the first light sourcetype and the reference white balance gain set for the second lightsource type and the mixture ratio acquisition unit acquires theinfluence rate of the first light source type and the second lightsource type from the first image data and the second image data andacquires the mixture ratio of the reference white balance gain set forthe first light source type and the reference white balance gain set forthe second light source type on the basis of the influence rate.Preferably, the recording unit stores the reference white balance gainset for the first light source type, the reference white balance gainset for the second light source type, and the mixture ratio in thestorage medium so as to be associated with the original image data.

According to this aspect, it is possible to perform multi-area WBprocessing considering the influence rate of the first light source typeand the second light source type subsequently.

Preferably, regardless of a white balance setting mode when the originalimage data is acquired, the light source type determination unitdetermines the number of light sources of the original image data andthe types of light source, the gain acquisition unit acquires thereference white balance gain set for each light source type, the mixtureratio acquisition unit acquires the influence rate of each light sourcetype for each pixel of the original image data and acquires the mixtureratio of the reference white balance gains on the basis of the influencerate, and the recording unit stores the reference white balance gainsand the mixture ratio in the storage medium so as to be associated withthe original image data.

According to this aspect, the reference white balance gains and themixture ratio are stored in the storage medium so as to be associatedwith the original image data, regardless of the white balance settingmode when the original image data is acquired. Therefore, the user canperform multi-area WB processing for the original image datasubsequently, regardless of the white balance setting mode when theoriginal image data is acquired.

Preferably, the image processing device further comprises a setting modedetermination unit that determines a white balance setting mode when theoriginal image data is acquired. Preferably, only in a case in which thewhite balance setting mode determined by the setting mode determinationunit is an auto white balance mode in which a white balance gain appliedto the original image data is determined on the basis of colordistribution information of the original image data, the light sourcetype determination unit determines the number of light sources of theoriginal image data and the types of light source, the gain acquisitionunit acquires the reference white balance gain set for each light sourcetype, the mixture ratio acquisition unit acquires the influence rate ofeach light source type for each pixel of the original image data andacquires the mixture ratio of the reference white balance gains on thebasis of the influence rate, and the recording unit stores the referencewhite balance gains and the mixture ratio in the storage medium so as tobe associated with the original image data.

According to this aspect, only in a case in which the white balancesetting mode when the original image data is acquired is the auto whitebalance mode, it is possible to perform multi-area WB processing for theoriginal image data subsequently.

The “auto white balance mode” is a mode in which the original image datais analyzed to determine a white balance gain and a method fordetermining the white balance gain is not particularly limited.

Another aspect of the invention relates to an image processing devicecomprising: a processed data acquisition unit that acquires originalimage data, a reference white balance gain which is set for each lightsource type of the original image data, and a mixture ratio of thereference white balance gains set for each pixel of the original imagedata from a storage medium; and a gain calculation unit that calculatesan application white balance gain for each pixel of the original imagedata from the reference white balance gains according to the mixtureratio.

According to this aspect, it is possible to accurately calculate theapplication white balance gain applied to the original image data.Therefore, it is possible to perform multi-area WB processing for theoriginal image data subsequently.

Preferably, the reference white balance gains acquired by the processeddata acquisition unit include a reference white balance gain for flashlight which is set in a case in which the light source type is flashlight and a reference white balance gain for environmental light whichis set in a case in which the light source type is environmental lightand the mixture ratio acquired by the processed data acquisition unit isa mixture ratio of the reference white balance gain for flash light andthe reference white balance gain for environmental light which are setfor each pixel of the original image data. Preferably, the gaincalculation unit calculates the application white balance gain for eachpixel of the original image data from the reference white balance gainfor flash light and the reference white balance gain for environmentallight according to the mixture ratio.

According to this aspect, it is possible to perform multi-area WBprocessing considering the influence of flash light and environmentallight subsequently.

Preferably, the image processing device further comprises: a settingmode acquisition unit that acquires information about a white balancesetting mode when the original image data is acquired; and a processingmode acquisition unit that acquires information about a white balanceprocessing mode for the original image data. Preferably, the gaincalculation unit calculates the application white balance gain on thebasis of the reference white balance gain acquired from the storagemedium in a case in which the white balance processing mode isdetermined to be the same as the white balance setting mode on the basisof the information about the white balance setting mode and theinformation about the white balance processing mode, acquires thereference white balance gain which is set on the basis of the whitebalance processing mode in a case in which the white balance processingmode is determined to be different from the white balance setting mode,and calculates the application white balance gain on the basis of thereference white balance gain.

According to this aspect, even in a case in which the white balancesetting mode when the original image data is acquired is different fromthe white balance processing mode which is intended when multi-area WBprocessing is performed for the original image data, it is possible toperform multi-area WB processing in which the intended white balanceprocessing mode is reflected.

The “white balance processing mode” may be determined by any method orthe user may determine the white balance processing mode. Alternatively,the image processing device may determine the white balance processingmode on the basis of various conditions.

Preferably, the image processing device further comprises: a lightsource type designation unit that receives a light source typedesignated by a user; and a designated gain acquisition unit thatacquires a reference white balance gain set for the light source typedesignated by the user which is received by the light source typedesignation unit. Preferably, the gain calculation unit replaces atleast some of the reference white balance gains set for each lightsource type of the original image data with the reference white balancegain acquired by the designated gain acquisition unit and calculates theapplication white balance gain according to the mixture ratio.

According to this aspect, it is possible to perform multi-area WBprocessing in which the light source type designated by the user isreflected.

Preferably, the image processing device further comprises a whitebalance processing unit that applies the application white balance gainto the original image data to acquire white-balance-adjusted image data.

According to this aspect, it is possible to acquire thewhite-balance-adjusted image data subjected to multi-area WB processing.

Preferably, the image processing device further comprises a displaycontrol unit and a display unit that is controlled by the displaycontrol unit. Preferably, the display control unit displays an imagebased on the white-balance-adjusted image data on the display unit.

According to this aspect, the user can simply check the image based onthe white-balance-adjusted image data through the display unit.

Preferably, the display control unit is connected to an operating unitthat is operated by the user, displays a light source display portionindicating the light source type of the original image data on thedisplay unit, and displays a plurality of light source display portionswhich are provided for each light source type on the display unit in acase in which there are a plurality of light source types in theoriginal image data. Preferably, in a case in which the plurality oflight source display portions are displayed on the display unit, whenthe user operates the operating unit to designate any one of theplurality of light source display portions, the display control unitdisplays a portion of or the entire image displayed on the display unitso as to be highlighted according to an influence rate of a light sourcecorresponding to the light source display portion designated through theoperating unit.

According to this aspect, the user can simply check the magnitude of theinfluence rate of a desired light source through the image that isdisplayed on the display unit so as to be highlighted.

Preferably, the display control unit displays a pixel, which is moreaffected by the light source corresponding to the light source displayportion that is designated by the user through the operating unit thanby other types of light sources among the light sources of the originalimage data, among pixels of the image on the display unit so as to behighlighted.

According to this aspect, the user can simply check an image portion,which is more affected by a desired light source than by other types oflight source, through the image that is displayed on the display unit soas to be highlighted.

Preferably, the display control unit displays a pixel, on which theinfluence rate of the light source corresponding to the light sourcedisplay portion that is designated by the user through the operatingunit is higher than a first rate, among the pixels of the image on thedisplay unit so as to be highlighted.

According to this aspect, the user can simply check an image portion, onwhich the influence rate of a desired light source is high, through theimage that is displayed on the display unit so as to be highlighted.

The “first rate” is not particularly limited. For example, the “firstrate” may be a predetermined value, such as “50%”, or may beappropriately set according to, for example, the number of light sourcesof the original image data and the types of light source.

Preferably, the image processing device further comprises a whitebalance adjustment unit that adjusts a white balance of the image.Preferably, the display control unit is connected to an operating unitthat is operated by the user, displays a light source display portionindicating the light source type of the original image data on thedisplay unit, and displays a plurality of light source display portionswhich are provided for each light source type on the display unit in acase in which there are a plurality of light source types in theoriginal image data. Preferably, in a case in which the plurality oflight source display portions are displayed on the display unit, whenthe user operates the operating unit to designate any one of theplurality of light source display portions, the display control unitdisplays a change display portion for receiving a change in the whitebalance based on the light source corresponding to the light sourcedisplay portion which is designated through the operating unit on thedisplay unit. Preferably, the display control unit receives the changein the white balance of the image through the change display portion inresponse to an operation of the user through the operating unit.Preferably, when the display control unit receives the change in thewhite balance, the white balance adjustment unit adjusts the whitebalance of the image and reflects the change in the image and thedisplay control unit displays the image of which the white balance hasbeen adjusted by the white balance adjustment unit on the display unit.

According to this aspect, the user can simply check the image, of whichthe white balance related to the light source designated by the userthrough the light source display portion displayed on the display unithas been changed and adjusted, through the display unit.

Preferably, the image processing device further comprises a whitebalance adjustment unit that adjusts a white balance of the image.Preferably, the display control unit is connected to an operating unitthat is operated by the user, displays a light source display portionindicating the light source type of the original image data on thedisplay unit, and displays a plurality of light source display portionswhich are provided for each light source type on the display unit in acase in which there are a plurality of light source types in theoriginal image data. Preferably, in a case in which the plurality oflight source display portions are displayed on the display unit, whenthe user operates the operating unit to designate any one of theplurality of light source display portions, the display control unitdisplays a change display portion for receiving a change in an influencerate of the light source corresponding to the light source displayportion which is designated through the operating unit in the image onthe display unit. Preferably, the display control unit receives anamount of change in the influence rate in the image through the changedisplay portion in response to an operation of the user through theoperating unit and the white balance adjustment unit adjusts the whitebalance of the image and reflects the amount of change in the influencerate which has been received by the display control unit through thechange display portion in the image. Preferably, the display controlunit displays the image of which the white balance has been adjusted bythe white balance adjustment unit on the display unit.

According to this aspect, the user can simply check the image, of whichthe white balance related to the light source designated by the user hasbeen changed and adjusted, through the display unit.

Preferably, the white balance setting mode is any one of a preset whitebalance mode in which the white balance gain is preset, an auto whitebalance mode in which the white balance gain applied to the originalimage data is determined on the basis of color distribution informationof the original image data, and a custom white balance mode in which thewhite balance gain applied to the original image data is determined onthe basis of color distribution information of reference image datawhich is different from the original image data.

Preferably, the original image data is RAW image data.

According to this aspect, for example, it is possible to acquirewhite-balance-adjusted image data when the original image data (RAWimage data) is developed.

Preferably, the original image data is uncompressed image data.

Preferably, the original image data is reversibly compressed image data.

Preferably, the original image data is irreversibly compressed imagedata.

It is possible to acquire white-balance-adjusted image data from theoriginal image data.

Still another aspect of the invention relates to an imaging devicecomprising an imaging element and the above-mentioned image processingdevice. The original image data is acquired by the imaging element.

Yet another aspect of the invention relates to an image processingmethod comprising: determining the number of light sources of originalimage data and the types of light source; acquiring a reference whitebalance gain which is set for each light source type of the originalimage data; acquiring an influence rate of each light source type foreach pixel of the original image data and acquiring a mixture ratio ofthe reference white balance gains on the basis of the influence rate;and storing the reference white balance gains and the mixture ratio in astorage medium so as to be associated with the original image data.

Still yet another aspect of the invention relates to an image processingmethod comprising: acquiring original image data, a reference whitebalance gain which is set for each light source type of the originalimage data, and a mixture ratio of the reference white balance gains setfor each pixel of the original image data from a storage medium; andcalculating an application white balance gain for each pixel of theoriginal image data from the reference white balance gains according tothe mixture ratio.

Yet still another aspect of the invention relates to a program thatcauses a computer to perform: a step of determining the number of lightsources of original image data and the types of light source; a step ofacquiring a reference white balance gain which is set for each lightsource type of the original image data; a step of acquiring an influencerate of each light source type for each pixel of the original image dataand acquiring a mixture ratio of the reference white balance gains onthe basis of the influence rate; and a step of storing the referencewhite balance gains and the mixture ratio in a storage medium so as tobe associated with the original image data.

Still yet another aspect of the invention relates to a program thatcauses a computer to perform: a step of acquiring original image data, areference white balance gain which is set for each light source type ofthe original image data, and a mixture ratio of the reference whitebalance gains set for each pixel of the original image data from astorage medium; and a step of calculating an application white balancegain for each pixel of the original image data from the reference whitebalance gains according to the mixture ratio.

According to the invention, it is possible to accurately performmulti-area WB processing for the original image data subsequently, usingthe reference white balance gains and the mixture ratio stored in thestorage medium. In addition, when multi-area WB processing is performedsubsequently, it is possible to adjust the reference white balance gainsand the mixture ratio. Therefore, it is possible to perform whitebalance adjustment with high flexibility.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a front perspective view illustrating a digital camera.

FIG. 2 is a rear perspective view illustrating the digital camera.

FIG. 3 is a block diagram illustrating a control processing system ofthe digital camera.

FIG. 4 is a block diagram illustrating an example of the functionalstructure of an image processing unit according to a first embodiment.

FIG. 5 is a flowchart illustrating the flow of image processingaccording to the first embodiment.

FIG. 6 is a block diagram illustrating an example of the functionalstructure of an image processing unit according to a second embodiment.

FIG. 7 is a flowchart illustrating the flow of image processingaccording to the second embodiment.

FIG. 8 is a block diagram illustrating an example of the functionalstructure of an image processing unit according to a third embodiment.

FIG. 9 is a flowchart illustrating the flow of image processingaccording to the third embodiment.

FIG. 10 is a flowchart illustrating the flow of image processingaccording to a fourth embodiment.

FIG. 11 is a conceptual diagram illustrating the connection among adigital camera, a computer, a server, and a portable terminal through anetwork.

FIG. 12 is a block diagram illustrating an example of the functionalstructure of a data processing unit (image processing unit) according toa fifth embodiment.

FIG. 13 is a flowchart illustrating the flow of image processingaccording to the fifth embodiment.

FIG. 14 is a block diagram illustrating an example of the functionalstructure of a data processing unit (image processing unit) according toa sixth embodiment.

FIG. 15 is a flowchart illustrating the flow of image processingaccording to the sixth embodiment.

FIG. 16 is a block diagram illustrating an example of the functionalstructure of a data processing unit (image processing unit) according toa seventh embodiment.

FIG. 17 is a flowchart illustrating the flow of image processingaccording to the seventh embodiment.

FIG. 18 is a block diagram illustrating an example of the functionalstructure of an image processing unit according to an eighth embodiment.

FIG. 19 is a diagram illustrating an example of the display of an imageon a display unit.

FIG. 20 is a diagram illustrating an example of the display of an imageon the display unit in a mode in which a user designates highlighting.

FIG. 21 is a diagram illustrating an example of the display of an imageon the display unit when the user designates highlighting.

FIG. 22 is a block diagram illustrating an example of the functionalstructure of an image processing unit according to a ninth embodiment.

FIG. 23 is a diagram illustrating an example of the display of an imageon a display unit according to the ninth embodiment.

FIG. 24 is a diagram illustrating the outward appearance of a smartphone.

FIG. 25 is a block diagram illustrating the structure of the smart phoneillustrated in FIG. 24.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the invention will be described with reference to thedrawings. In the following embodiments, an example in which theinvention is applied to a digital camera (imaging device) will bedescribed. However, the invention can be applied to an image processingdevice, an imaging device, an image processing method, and a program, inaddition to the digital camera.

FIG. 1 is a front perspective view illustrating a digital camera 2 andFIG. 2 is a rear perspective view illustrating the digital camera 2.

The digital camera 2 comprises a camera body 3 and a lens barrel 4 thatis attached to a front surface of the camera body 3. The lens barrel 4and the camera body 3 may be integrally provided or may be detachablyprovided as an interchangeable lens camera.

In addition to the lens barrel 4, a flash light emitting unit 5 isprovided on the front surface of the camera body 3. A shutter button 6and a power switch 7 are provided on an upper surface of the camera body3. The shutter button 6 is an imaging instruction unit that receives animaging instruction from a user. The power switch 7 is a power switchingunit that receives an instruction to turn on and off the digital camera2 from the user.

A display unit 8 which is, for example, a liquid crystal panel and anoperating unit 9 which is directly operated by the user are provided ona rear surface of the camera body 3. The display unit 8 displays a liveview image (through image) in an imaging standby state to function as anelectronic viewfinder and functions as a reproduction image display unitwhen a captured image or a memory-stored image is reproduced.

The operating unit 9 is an arbitrary operating device, such as a modeswitch, a cross key, and an execution key. For example, the mode switchis operated by the user to switch the operation mode of the digitalcamera 2. Examples of the operation mode of the digital camera 2 includean imaging mode in which an image of an object is captured to obtain acaptured image and a playback mode in which an image is played back anddisplayed. Examples of the imaging mode include an auto focus (AF) modein which auto focus is performed and a manual focus (MF) mode in which amanual focus operation is performed. The cross key and the execution keyare operated by the user to display a menu screen or a setting screen onthe display unit 8, to move a cursor displayed on the menu screen or thesetting screen, or to confirm various types of settings of the digitalcamera 2.

A memory slot into which a main memory 10 is inserted and a cover thatopens and closes an opening of the memory slot are provided at thebottom (not illustrated) of the camera body 3. The main memory 10 isdetachably provided in the camera body 3. When the main memory 10 isinserted into the camera body 3, it is electrically connected to astorage control unit 33 provided in the camera body 3. The main memory10 can be generally a semiconductor memory, such as a card-type flashmemory. The main memory 10 is not particularly limited. For example, arecording medium of an arbitrary storage type, such as a magneticmedium, can be used as the main memory 10.

FIG. 3 is a block diagram illustrating a control processing system ofthe digital camera 2.

Object light passes through a lens unit 12 that is provided in the lensbarrel 4 and a mechanical shutter 20 that is provided in the camera body3 and is received by an imaging element 21. The imaging element 21 is anelement that receives the object image and generates image data andincludes color filters, such as red, green, and blue (R, G, and B)filters, and an image sensor, such as a charge coupled device (CCD) or acomplementary metal oxide semiconductor (CMOS), which converts anoptical image into an electric signal. For example, an automatic gaincontrol (AGC) circuit of a process processing unit 22 performs processprocessing for image data that is output from the imaging element 21 andan A/D conversion unit 23 converts analog image data into digital imagedata. The digital image data is stored in a buffer memory 24.

The buffer memory 24 is an area that temporarily stores the image dataand is, for example, a dynamic random access memory (DRAM). The imagedata that has been transmitted from the A/D conversion unit 23 and thenstored in the buffer memory 24 is read by a data processing unit 31which is controlled by a system control unit 25. The data processingunit 31 performs various types of image processing, such as a gammacorrection process and a demosaic process, using the image datagenerated by the imaging element 21 as input image data, and stores theimage data subjected image processing in the buffer memory 24 again.

The image data which has been subjected to the image processing by thedata processing unit 31 and then stored in the buffer memory 24 is readby a display control unit 35 and a compression and decompression unit32. The display control unit 35 controls the display unit 8 such thatthe image data read from the buffer memory 24 is displayed on thedisplay unit 8. As such, the image data which has been output from theimaging element 21 and then subjected to the image processing by thedata processing unit 31 is displayed as an imaging check image(post-view image) on the display unit 8.

The compression and decompression unit 32 compresses the image data readfrom the buffer memory 24 to create image data with an arbitrarycompression format, such as Joint Photographic Experts Group (JPEG) orTagged Image File Format (TIFF). The compressed image data is stored inthe main memory 10 by the storage control unit 33 that controls aprocess of storing data in the main memory (storage medium) 10 and aprocess of reading data from the main memory 10. In a case in which adata type, such as image data, is stored in the main memory 10, thestorage control unit 33 adds imaging information, such as editing dateand time information (update date and time information), or other kindsof related information to the data type, on the basis of date and timeinformation acquired from a clock device 34 which will be describedbelow. The imaging information is added to the image data in any format.For example, an exchangeable image file format (Exif) can be used.

In the playback mode in which the image data stored in the main memory10 is played back, the image data stored in the main memory 10 is readby the storage control unit 33 that is controlled by the system controlunit 25, is decompressed by the compression and decompression unit 32,and is then stored in the buffer memory 24. The image data is read fromthe buffer memory 24 by the display control unit 35 and is played backand displayed on the display unit 8 in the same order as that in which acaptured image is checked and displayed.

As described above, the system control unit 25 controls the buffermemory 24, the data processing unit 31, and the storage control unit 33.In addition, the system control unit 25 controls other units in thedigital camera 2. For example, the system control unit 25 controls alens driving unit 27 such that the driving of the lens unit 12 iscontrolled and controls a shutter driving unit 26 such that the drivingof the mechanical shutter 20 is controlled. In addition, the systemcontrol unit 25 controls the imaging element 21 such that the output ofthe image data is controlled. Further, the system control unit 25controls the flash light emitting unit 5 such that the emission ornon-emission of flash light is controlled, and controls a power controlunit 28 such that, for example, whether a battery is mounted on a powersupply 29, the type of battery, and a remaining battery level aredetected. Furthermore, the system control unit 25 acquires the date andtime information which is counted by the clock device 34 and uses thedate and time information in various types of processes. In addition,the system control unit 25 controls various processing units forming thedata processing unit 31.

The system control unit 25 acquires an operation signal from a userinterface 36 including the shutter button 6, the power switch 7, and theoperating unit 9 and performs various types of processes and devicecontrol corresponding to the operation signal. For example, the systemcontrol unit 25 controls the shutter driving unit 26 such that theopening and closing of the mechanical shutter 20 is controlled inresponse to a release signal received from the shutter button 6.Furthermore, the system control unit 25 controls the power control unit28 such that the turn-on and tune-off of the power supply 29 arecontrolled in response to a power on/off signal received from the powerswitch 7.

Programs or data types required for various types of processes anddevice control performed by the system control unit 25 are stored in acontrol memory 30. The system control unit 25 can read the programs orthe data types stored in the control memory 30, if necessary. Inaddition, the system control unit 25 can store a new program or a newdata type in the control memory 30. For example, the system control unit25 can write condition data, such as the type of set white balance mode(hereinafter, referred to as a “WB mode”) or a white balance gain, tothe control memory 30. The system control unit 25 can control thedisplay control unit 35 such that various kinds of information acquiredfrom each unit are displayed on the display unit 8. In addition, thesystem control unit 25 can change various kinds of information to bedisplayed on the display unit 8, in response to an operation signalwhich is input from the user through the user interface 36.

Next, for white balance processing for the original image data acquiredby the imaging element 21, an example of image processing will bedescribed.

In each of the following embodiments, an example in which “RAW imagedata” that has been output from the imaging element 21 and passedthrough the process processing unit 22 (process processing) and the A/Dconversion unit 23 (A/D conversion process) is input to the dataprocessing unit 31 through the buffer memory 24 and is used as originalimage data will be described. Therefore, the demosaic process of thedata processing unit 31 for the following original image data (RAW imagedata) is skipped. The original image data is not limited to the “RAWimage data” and the same image processing can be performed for imagedata with any other formats, such as a JPEG format and TIFF.

In general, there is a type of digital camera that can store RAW imagedata which is “image data before image quality processing” or “imagedata subjected to only image quality processing (for example, lensshading correction) caused by an imaging system” in a storage medium(main memory 10), instead of or in addition to a processed image such asa JPEG image subjected to image quality processing. The storage of theRAW image data makes it possible for the user to perform image qualityprocessing and a development process for the RAW image datasubsequently, using RAW development software, and to obtain images withdifferent image qualities, without taking a picture again. In a case inwhich the RAW image data is stored in the storage medium, the digitalcamera also stores information (for example, information about imagingconditions and the type of digital camera) required to develop the RAWimage data in the storage medium. A method for storing the “RAW imagedata” and “information data required for development” is notparticularly limited. Data in which information data required fordevelopment or other information is added as tag information (meta data)to the “RAW image data” may be stored in the storage medium. In thefollowing embodiments, data in which tag information (meta data) isadded to RAW image data is referred to as “RAW data”.

A compression format of the original image data to be stored in the mainmemory 10 is not particularly limited. The original image data may beuncompressed image data, reversibly-compressed image data, orirreversibly-compressed image data. In a case in which the originalimage data is RAW image data, a development process may be performedbefore WB processing (see a “white balance processing unit (WBprocessing unit) 64” which will be described below) or after the WBprocessing. In a case in which the original image data is compressedimage data (reversibly-compressed image data or irreversibly-compressedimage data), it is preferable that a decompression process is performedbefore WB processing (see the “WB processing unit 64” which will bedescribed below). The development process and the decompression processmay be performed by, for example, a processed data acquisition unit 60which will be described below.

First Embodiment

This embodiment relates to a processing structure for implementingmulti-area WB processing that mixes a plurality of WB gains (referencewhite balance gains (hereinafter, referred to as “reference WB gains”))at different mixture ratios for each pixel to acquire a WB gain(application white balance gain (hereinafter, referred to as an“application WB gain”)) suitable for each pixel. In particular, in thisembodiment, the main memory (storage medium) 10 does not store the “WBgain (application WB gain) suitable for each pixel”, but stores “aplurality of WB gains (reference WB gains)” used to acquire the “WB gain(application WB gain) suitable for each pixel” and a “mixture ratio foreach pixel”.

FIG. 4 is a block diagram illustrating an example of the functionalstructure of an image processing unit 38 according to a firstembodiment.

The image processing unit 38 according to this embodiment includes adata processing unit 31 and a storage control unit 33. The dataprocessing unit 31 includes a light source type determination unit 40, again acquisition unit 42, and a mixture ratio acquisition unit 44.

The light source type determination unit 40 determines the number oflight sources of the original image data and the types of light source.A method for determining the number of light sources of the originalimage data and the types of light source is not particularly limited.for example, in a case in which flash light and environmental light areassumed as the light source type, the light source type determinationunit 40 may acquire information about whether flash light is emittedfrom the system control unit 25 that controls the flash light emittingunit 5 to determine the number of light sources of the original imagedata and the types of light source. Whether flash light is emitted isset by the user through the user interface 36 or is determined by thesystem control unit 25 on the basis of the detection result of aphotometric sensor (not illustrated) that detects the brightness of acaptured scene. In both cases, the system control unit 25 hasinformation about whether flash light is emitted. Therefore, in a casein which environmental light and flash light are assumed as the lightsource type, the light source type determination unit 40 may acquire theinformation about whether flash light is emitted from the system controlunit 25 and determine the number of light sources of the original imagedata and the types of light source.

The gain acquisition unit 42 acquires a reference WB gain that is setfor each light source type of the original image data. For example, forthe light source type of flash light, the WB gain related to the flashlight can be predetermined and stored in a memory such as the controlmemory 30. The gain acquisition unit 42 may acquire the predetermined WBgain for flash light as the reference WB gain from the memory. For thelight source type of environmental light, the gain acquisition unit 42may analyze, for example, the original image data to acquire colordistribution information or brightness distribution information,estimate a WB gain suitable for the environmental light from the colordistribution information or the brightness distribution information, andacquire the WB gain as the reference WB gain.

The mixture ratio acquisition unit 44 acquire the influence rate of eachlight source type for each pixel of the original image data and acquiresthe mixture ratio of the reference WB gains on the basis of theinfluence rate. Here, the “influence rate” is calculated by any method.For example, the original image data can be analyzed to calculate the“influence rate”. In a second embodiment which will be described below,the influence rate is acquired on the basis of a “brightness value”obtained by analyzing image data and the mixture ratio of the referenceWB gains is acquired on the basis of the influence rate.

The storage control unit 33 functions as a recording unit that storesthe reference WB gains and the mixture ratio in the main memory (storagemedium) 10 so as to be associated with the original image data. Theoriginal image data is output from the mixture ratio acquisition unit 44(data processing unit 31), is transmitted to the storage control unit 33through the buffer memory 24 and the compression and decompression unit32 (see FIG. 3), and is stored in the main memory 10. In a case in whichthe original image data to be stored in the main memory 10 is compressedimage data (reversibly compressed image data or irreversibly compressedimage data), the compression and decompression unit 32 performs acompression process corresponding to a compression format. In a case inwhich the original image data is uncompressed image data, thecompression process of the compression and decompression unit 32 isskipped.

FIG. 5 is a flowchart illustrating the flow of image processingaccording to the first embodiment. In the following description of theflowchart, for example, a case in which image data acquired while flashlight and environmental light are emitted to an object is used as theoriginal image data will be described.

First, the light source type determination unit 40 determines the numberof light sources of the original image data and the types of lightsource (S11 in FIG. 5). In this example, flash light and environmentallight are determined as two light source types.

Then, the gain acquisition unit 42 acquires a “WB gain WB1 suitable forenvironmental light (hereinafter, referred to as a “reference WB gainWB1 for environmental light”)” which is set in a case in which the lightsource type is environmental light and a “WB gain WB2 suitable for flashlight (hereinafter, referred to as a “reference WB gain WB2 for flashlight”)” which is set in a case in which the light source type is flashlight (S12).

Then, the mixture ratio acquisition unit 44 acquires the influence rateof each of the flash light and the environmental light for each pixel ofthe original image data and acquires a mixture ratio “Ratio_ij (where“i” indicates an X-axis value and “j” indicates a Y-axis value in a casein which a pixel position is represented by the X-axis value and theY-axis value) of the reference WB gain for flash light and the referenceWB gain for environmental light on the basis of the influence rate(S13).

Then, the storage control unit 33 stores the reference WB gain WB1 forenvironmental light, the reference WB gain WB2 for flash light, and themixture ratio Ratio_ij in the main memory 10 so as to be associated withthe original image data (S14).

As described above, according to this embodiment, the “reference WB gainfor each light source type” and the “mixture ratio” which are the basisfor calculating the WB gain used in multi-area WB processing are storedso as to be associated with the original image data. Therefore, it ispossible to simply acquire the WB gain suitable for each pixel, usingthe stored “reference WB gain for each light source type” and “mixtureratio”. As a result, it is possible to perform multi-area WB processingfor the original image data subsequently in response to requests fromthe user. For example, RAW image data is stored as the original imagedata in the main memory 10 together with the “reference WB gain for eachlight source type” and the “mixture ratio”. In this case, whenperforming a development process for the RAW image data, the user candetermine whether to perform the multi-area WB processing.

In a case in which the multi-area WB processing is performedsubsequently, it is possible to change the “reference WB gain for lightsource type” and/or the “mixture ratio” in response to requests from theuser and thus to improve flexibility in white balance adjustment.Therefore, for example, the user can individually adjust the whitebalances of an “image portion such as a person image with a highinfluence rate of flash light” and an “image portion such as abackground image with a low influence rate of flash light”.

Second Embodiment

In this embodiment, the detailed description of the same or similarstructure and operation as those in the first embodiment will not berepeated.

This embodiment relates to a detailed aspect of the first embodiment andrelates a case in which flash emission image data obtained by capturinga scene while emitting flash light is used as the original image data.

FIG. 6 is a block diagram illustrating an example of the functionalstructure of an image processing unit 38 according to a secondembodiment.

A data processing unit 31 (image processing unit 38) according to thisembodiment includes a flash image acquisition unit 46, in addition tothe light source type determination unit 40, the gain acquisition unit42, and the mixture ratio acquisition unit 44. The flash imageacquisition unit 46 acquires flash emission image data which is capturedwhile flash light is emitted and flash non-emission image data which isobtained by capturing an image of the same object as that in the flashemission image data, when flash light is not emitted. That is, in thisembodiment, when the user presses the shutter button 6 (see FIGS. 1 and2) in order to acquire the flash emission image data, the system controlunit 25 controls, for example, the imaging element 21, while controllingthe emission of light from the flash light emitting unit 5, such that animaging process using flash light and an imaging process without usingflash light are automatically and continuously performed. The flashimage acquisition unit 46 acquires the flash emission image data and theflash non-emission image data captured by these imaging processes.

The original image data to be processed which is stored in the mainmemory 10 is flash emission image data. However, flash non-emissionimage data may be stored in the main memory 10 together with the flashemission image data.

The flash emission image data and the flash non-emission image data aretransmitted from the flash image acquisition unit 46 to the mixtureratio acquisition unit 44 and the flash non-emission image data is alsotransmitted to the gain acquisition unit 42.

The flash non-emission image data is image data which is not affected byflash light and is captured under environmental light as a main lightsource. Therefore, the gain acquisition unit 42 analyzes the flashnon-emission image data to acquire the reference WB gain forenvironmental light. Specifically, the gain acquisition unit 42 maycalculate a WB gain for an auto white balance mode (hereinafter,referred to as an “AWB gain”) and set the AWB gain as the reference WBgain for environmental light. The AWB gain can be calculated by anymethod. For example, the gain acquisition unit 42 may analyze the flashnon-emission image data to obtain color distribution information from R,G, and B pixel values and set a gain at which the mean is an achromaticcolor as the AWB gain (WB gain for environmental light). In addition,the gain acquisition unit 42 may obtain brightness distributioninformation from the brightness value of the flash non-emission imagedata and determine the AWB gain (WB gain for environmental light) on thebasis of the brightness distribution information. Furthermore, the WBgain which is set for each of a plurality of environmental light typesmay be stored in a memory, such as the control memory 30, in advance andthe gain acquisition unit 42 may analyze the flash non-emission imagedata to determine the type of environmental light, acquire the WB gainrelated to the determined type of environmental light from the memory,and set the WB gain as the WB gain for environmental light.

In contrast, the reference WB gain for flash light is calculated inadvance on the basis of the characteristics of flash light and is thenstored in a memory such as the control memory 30. The gain acquisitionunit 42 reads the reference WB gain for flash light from the memory andacquires the reference WB gain for flash light.

The mixture ratio acquisition unit 44 acquires the influence rate ofeach of the flash light and the environmental light from the flashemission image data and the flash non-emission image data acquired bythe flash image acquisition unit 46 and acquires the mixture ratio ofthe reference white balance gain for flash light and the reference whitebalance gain for environmental light on the basis of the influence rate.

Specifically, in this example, the mixture ratio acquisition unit 44acquires the brightness value (hereinafter, referred to as a “firstbrightness value”) of each pixel of the flash non-emission image data,acquires the brightness value (hereinafter, referred to as a “secondbrightness value”) of each pixel of the flash emission image data, andacquires the influence rate of each of the flash light and theenvironmental light on the basis of the first brightness value and thesecond brightness value.

A method for acquiring the first brightness value and the secondbrightness value is not particularly limited. For example, in each ofthe flash emission image data and the flash non-emission image data, ina case in which an R pixel value is represented by “R_ij (where “i”indicates an X-axis value and “j” indicates a Y-axis value in a case inwhich a pixel position is represented by the X-axis value and the Y-axisvalue)“, a G pixel value is represented by “G_ij (where “i” indicates anX-axis value and “j” indicates a Y-axis value in a case in which a pixelposition is represented by the X-axis value and the Y-axis value)”, anda B pixel value is represented by “B_ij (where “i” indicates an X-axisvalue and “j” indicates a Y-axis value in a case in which a pixelposition is represented by the X-axis value and the Y-axis value)”, thebrightness value (first brightness value) “Y1_ij (where “i” indicates anX-axis value and “j” indicates a Y-axis value in a case in which a pixelposition is represented by the X-axis value and the Y-axis value)” ofeach pixel of the flash non-emission image data and the brightness value(second brightness value) “Y2_ij (where “i” indicates an X-axis valueand “j” indicates a Y-axis value in a case in which a pixel position isrepresented by the X-axis value and the Y-axis value)” of each pixel ofthe flash emission image data can be represented by the followingExpressions 1 and 2. In the following Expressions 1 and 2, “0.299”,“0.587”, and “0.114” are used as coefficients. However, othercoefficients may be used.

Y1_ij=R_ij×0.299+G_ij×0.587+B_ij×0.114   [Expression 1]

Y2_ij=R_ij×0.299+G_ij×0.587+B_ij×0.114   [Expression 2]

The mixture ratio acquisition unit 44 can acquire a flash influence rate“Flash_ij” and a mixture ratio “Ratio_ij” on the basis of the firstbrightness value Y1_ij and the second brightness value Y2_ij, using thefollowing Expression 3.

Ratio_ij=Flash_ij=1−(Y1_ij/Y2_ij)   [Expression 3]

Similarly to the first embodiment, the storage control unit 33 storesthe acquired reference WB gain for environmental light, reference WBgain for flash light, and mixture ratio in the main memory 10 so as tobe associated with the original image data (flash emission image data).The mixture ratio Ratio_ij may be equivalent to the flash influence rateFlash_ij as in this example (see Expression 3), may be different fromthe flash influence rate Flash_ij, or may be a value in which the flashinfluence rate Flash_ij is reflected.

FIG. 7 is a flowchart illustrating the flow of image processingaccording to the second embodiment.

First, the light source type determination unit 40 determines that thenumber of light source types of the original image data is two (flashlight and environmental light) in (S21 in FIG. 7) and the flash imageacquisition unit 46 acquires flash emission image data and flashnon-emission image data (S22).

Then, the gain acquisition unit 42 acquires the reference WB gain WB1for environmental light from the flash non-emission image data (S23) andacquires the reference WB gain WB2 for flash light from the memory(S24).

Then, the mixture ratio acquisition unit 44 acquires the brightnessvalue (first brightness value) Y1_ij of each pixel of the flashnon-emission image data (S25) and acquires the brightness value (secondbrightness value) Y2_ij of each pixel of the flash emission image data(S26). Then, the mixture ratio acquisition unit 44 acquires the flashinfluence rate Flash_ij for each pixel of the original image data (flashemission image data) from the first brightness value Y1_ij and thesecond brightness value Y2_ij and sets the mixture ratio Ratio_ij on thebasis of the flash influence rate (S27).

Then, the storage control unit 33 stores the reference WB gain WB1 forenvironmental light, the reference WB gain WB2 for flash light, and themixture ratio Ratio_ij in the main memory 10 so as to be associated withthe original image data (flash emission image data) (S28).

As described above, according to this embodiment, the “reference WB gainfor environmental light”, the “reference WB gain for flash light”, andthe “mixture ratio” which are the basis for calculating the WB gain setfor each pixel of the original image data (flash emission image data)acquired by an imaging process with flash light are stored so as to beassociated with the original image data. Therefore, in a case in whichthe white balance of an image is adjusted subsequently, it is possibleto individually adjust the white balance of a “image portion dominantlyaffected by flash light” and an “image portion dominantly affected byenvironmental light”, using the stored “reference WB gain forenvironmental light”, “reference WB gain for flash light”, and “mixtureratio”.

For example, in a case in which the user “wants to slightly correct onlya person image portion greatly affected by flash light to the red side,the reference WB gain WB2 for flash light suitable for flash light isslightly shifted to the red side. In this case, only the color (colorbalance) of the pixel having a high proportion of the reference WB gainWB2 for flash light is shifted to the red side and the color (colorbalance) of the pixel having a low proportion of the reference WB gainWB2 for flash light and a high proportion of the reference WB gain WB1for environmental light is less likely to be affected. As a result, thecolor of a person image portion having a high proportion of thereference WB gain WB2 for flash light is corrected to the red side andthe color of a background image portion having a high proportion of thereference WB gain WB1 for environmental light is maintained. Therefore,it is possible to perform white balance adjustment that meets the demandof the user.

In the second embodiment illustrated in FIGS. 6 and 7, an example inwhich flash light and environmental light are treated as the first lightsource type and the second light source type of the original image data,respectively, has been described. However, the first light source typeand the second light source type are not particularly limited. The lightsource type data acquisition unit (see FIG. 6) may acquire first imagedata indicating the influence of the first light source type and secondimage data indicating the influence of the second light source type,instead of the flash emission image data and the flash non-emissionimage data. That is, in a case in which the light source types of theoriginal image data determined by the light source type determinationunit include the first light source type and the second light sourcetype, the gain acquisition unit 42 may acquire a reference white balancegain that is set for the first light source type and is suitable for thefirst light source type and a reference white balance gain that is setfor the second light source type and is suitable for the second lightsource type. In this case, the mixture ratio acquisition unit 44acquires the influence rate of the first light source type and thesecond light source type of the original image data from the first imagedata and the second image data and acquires the mixture ratio of thereference white balance gain set for the first light source type and thereference white balance gain set for the second light source type, onthe basis of the influence rate. Then, the storage control unit 33 canstore the reference white balance gain set for the first light sourcetype, the reference white balance gain set for the second light sourcetype, and the mixture ratio in a storage medium so as to be associatedwith the original image data.

Third Embodiment

In this embodiment, the detailed description of the same or similarstructure and operation as those in the first and second embodimentswill not be repeated.

In this embodiment, the process of acquiring the reference WB gain andthe mixture ratio and the storage process in the first embodiment areperformed only in a case in which a white balance setting mode(hereinafter, referred to as a “WB setting mode”) when the originalimage data is acquired is an AWB mode. In a case in which the WB settingmode is the modes other than the AWB mode, such as a preset whitebalance mode (hereinafter, referred to as a “preset WB mode”), a customwhite balance mode (hereinafter, referred to as a “custom WB mode”), anda manual white balance mode (hereinafter, referred to as a “manual WBmode”), the process of acquiring the reference WB gain and the mixtureratio and the storage process in the first embodiment are not performed.

The WB setting mode which can be selected when the original image datais acquired is not particularly limited. For example, only apredetermined type of WB mode may be selected as the WB setting mode ora mode designated by the user or a mode corresponding to imagingconditions among a plurality of types of WB modes may be selected as theWB setting mode.

Therefore, for example, any one of the preset WB mode in which the WBgain is preset according to a predetermined light source type (forexample, daylight (sunlight), a fluorescent light, or a light bulb), theAWB mode in which the WB gain applied to the original image data isdetermined on the basis of the color distribution information of theoriginal image data, and the custom WB mode in which the WB gain appliedto the original image data is determined on the basis of the colordistribution information of reference image data different from theoriginal image data may be selected as the WB setting mode. In addition,the manual WB mode in which the user manually determines an individualWB gain may be selected as the WB setting mode.

Next, an example in which the same process as that in the firstembodiment is performed in a case in which the WB setting mode is theAWB mode will be described. However, the invention is not limitedthereto. For example, instead of the same process as that in the firstembodiment, the same process as that in the second embodiment will bedescribed.

FIG. 8 is a block diagram illustrating an example of the functionalstructure of an image processing unit 38 according to the thirdembodiment.

A data processing unit 31 (image processing unit 38) according to thisembodiment includes a setting mode determination unit 48, in addition tothe light source type determination unit 40, the gain acquisition unit42, and the mixture ratio acquisition unit 44. The setting modedetermination unit 48 determines the WB setting mode when the originalimage data is acquired. A method for determining the WB setting mode isnot particularly limited. In a case in which the system control unit 25has information about the WB setting mode, the setting modedetermination unit 48 may acquire the information about the WB settingmode from the system control unit 25.

In the example illustrated in FIG. 8, the setting mode determinationunit 48 and the system control unit 25 are separately illustrated.However, the system control unit 25 may function as the setting modedetermination unit 48. In this case, the system control unit 25 isincluded in the image processing unit 38.

The setting mode determination unit 48, the light source typedetermination unit 40, the gain acquisition unit 42, and the mixtureratio acquisition unit 44 included in the data processing unit 31 arecontrolled by the system control unit 25 and the system control unit 25can determine whether to perform the processes of each unit. Inparticular, in this embodiment, only in a case in which the WB settingmode determined by the setting mode determination unit 48 is the AWBmode, the image processing unit 38 performs the same process as that inthe first embodiment.

That is, only in a case in which the WB setting mode is the AWB mode,the light source type determination unit 40 determines the number oflight sources of the original image data and the types of light sourceand the gain acquisition unit 42 acquires the reference white balancegain set for each light source type. In addition, the mixture ratioacquisition unit 44 acquires the influence rate of each light sourcetype for each pixel of the original image data and acquires the mixtureratio of the reference white balance gains on the basis of the influencerate. Then, the storage control unit 33 stores the reference whitebalance gain and the mixture ratio in the main memory 10 so as to beassociated with the original image data.

FIG. 9 is a flowchart illustrating the flow of image processingaccording to the third embodiment. In the following description, anexample in which flash emission image data having flash light andenvironmental light as the light source type is used as the originalimage data will be described.

First, the setting mode determination unit 48 determines the WB settingmode when the original image data is acquired (S31 in FIG. 9) and thesystem control unit 25 determines whether the WB setting mode is the AWBmode (S32).

In a case in which the WB setting mode is determined to be the AWB mode(Y in S32), the light source type determination unit 40 determines thenumber of light sources of the original image data and the types oflight source (for example, flash light and environmental light) (S33).Then, the gain acquisition unit 42 acquires the reference white balancegain (the reference WB gain for environmental light and the reference WBgain for flash light) set for each light source type (S34).

Then, the mixture ratio acquisition unit 44 acquires the influence rateof each light source type for each pixel of the original image data andacquires the mixture ratio of the reference white balance gains (thereference WB gain for environmental light and the reference WB gain forflash light) on the basis of the influence rate (S35). Then, the storagecontrol unit 33 stores the reference white balance gains (the referenceWB gain for environmental light and the reference WB gain for flashlight) and the mixture ratio in the main memory 10 so as to beassociated with the original image data (S36).

On the other hand, in a case in which the WB setting mode is determinednot to be the AWB mode (N in S32), Steps S33 to S36 are skipped.

As described above, according to this embodiment, the reference whitebalance gain and the mixture ratio are stored in the main memory 10 soas to be associated with the original image data acquired in the AWBmode. Therefore, similarly to the first embodiment, the user can adjustthe white balance of an image subsequently with high flexibility. Forthe original image data acquired in the modes other than the AWB mode,Steps S33 to S36 are skipped. Therefore, it is possible to perform theprocess from the capture of the original image data to the storage ofthe original image data in the main memory 10 at a high speed.

Fourth Embodiment

In this embodiment, the detailed description of the same or similarstructure and operation as those in the first to third embodiments willnot be repeated.

In the third embodiment, even in a case in which the WB setting mode isthe AWB mode, the same process as that in the first embodiment or thesecond embodiment is performed. However, in this embodiment, the sameprocess as that in the first embodiment or the second embodiment isperformed regardless of the WB setting mode when the original image datais acquired.

As described above, a special process or information is required toperform multi-area WB processing. In a case in which the user wants toperform the multi-area WB processing when the original image data iscaptured and acquired, it is possible to perform a special process forthe multi-area WB processing or to acquire special information when theoriginal image data is acquired. However, in a case in which the userwants to perform the multi-area WB processing in a stage in which, forexample, RAW image data is developed after imaging, a special processfor the multi-area WB processing which is required during imaging is notperformed and information which needs to be acquired during imaging isnot acquired. Therefore, it is difficult to achieve the multi-area WBprocessing.

For example, in a case in which the multi-area WB processing isperformed for image data, which has been captured while flash light isemitted in a night portrait scene, as the original image data, it isnecessary to also acquire flash non-emission image data, to acquire theinfluence rate of flash for each pixel from the difference between thepixel values of the flash emission image data and the flash non-emissionimage data, and to acquire the mixture ratio of the reference WB gainfor environmental light and the reference WB gain for flash light foreach pixel according to the influence rate of flash. However, even ifthese processes are performed in the stage in which RAW image data(original image data) is developed, it is difficult to acquire the “WBgain 5suitable for each pixel” for performing multi-area WB processingsince only the flash emission image data is stored as the RAW image dataand the flash non-emission image data required for multi-area WBprocessing is absent.

Therefore, according to this modification example in which the referencewhite balance gain and the mixture ratio are stored in the main memory10 as in the first embodiment or the second embodiment, regardless ofthe WB setting mode when the original image data is acquired, even ifthe user wants to perform multi-area WB processing subsequently, it ispossible to acquire white-balance-adjusted image data (hereinafter,referred to as “WB-adjusted image data”) subjected to appropriatemulti-area WB processing.

The image processing unit 38 according to this embodiment has the samebasic structure as that according to the first embodiment (see FIG. 4)or the second embodiment (see FIG. 6). That is, the light source typedetermination unit 40 determines the number of light sources of theoriginal image data and the types of light source, regardless of the WBsetting mode when the original image data is acquired. The gainacquisition unit 42 acquires the reference white balance gain set foreach light source type. The mixture ratio acquisition unit 44 acquiresthe influence rate of each light source type for each pixel of theoriginal image data and acquires the mixture ratio of the referencewhite balance gains on the basis of the influence rate. The storagecontrol unit 33 stores the reference white balance gains and the mixtureratio in the main memory 10 so as to be associated with the originalimage data.

FIG. 10 is a flowchart illustrating the flow of image processingaccording to the fourth embodiment. In this embodiment, Steps S31 andS32 (see FIG. 9) related to the determination of the WB setting mode areomitted and Steps S33 to S36 are performed in the processing flow of thethird embodiment.

That is, the light source type determination unit 40 determines thenumber of light sources of the original image data and the types oflight source (for example, flash light and environmental light),regardless of the WB setting mode when the original image data isacquired (S33). Then, the gain acquisition unit 42 acquires thereference white balance gains (for example, the reference WB gain forenvironmental light and the reference WB gain for flash light) set foreach light source type (S34).

Then, the mixture ratio acquisition unit 44 acquires the influence rateof each light source type for each pixel of the original image data andacquires the mixture ratio of the reference white balance gains (forexample, the reference WB gain for environmental light and the referenceWB gain for flash light) on the basis of the influence rate (S35). Then,the storage control unit 33 stores the reference white balance gains(for example, the reference WB gain for environmental light and thereference WB gain for flash light) and the mixture ratio in the mainmemory 10 so as to be associated with the original image data (S36).

As described above, according to this embodiment, an operation formulti-area WB processing is always performed, regardless of the WBsetting mode when the original image data is acquired, and the referencewhite balance gains and the mixture ratio are stored in the main memory10 so as to be associated with the original image data. Therefore, theuser can determine whether or not to perform multi-area WB processingfor the original image data subsequently, regardless of the WB settingmode when the original image data is acquired.

Fifth Embodiment

In this embodiment, the detailed description of the same or similarstructure and operation as those in the first to fourth embodiments willnot be repeated.

This embodiment relates to a process which reads the reference WB gainand the mixture ratio from the main memory 10, in addition to theoriginal image data, and performs multi-area WB processing for theoriginal image data to obtain WB-adjusted image data.

That is, a data processing unit 31 according to this embodiment acquiresthe reference WB gains and the mixture ratio which are stored so as tobe associated with the original image data from the main memory 10 andperforms WB processing for the original image data. Therefore, thereference WB gains and the mixture ratio are stored in the main memory10 used in this embodiment so as to be associated with the originalimage data, for example, like the original image data according to thefirst to fourth embodiments.

The data processing unit 31 (see FIG. 12) according to this embodimentwhich will be described below may be provided integrally with orseparately from the data processing unit 31 (see FIGS. 4, 6, and 8)including the “light source type determination unit 40, the gainacquisition unit 42, and the mixture ratio acquisition unit 44”described in the first to third embodiments. Therefore, for example, ina case in which the data processing unit 31 illustrated in FIGS. 4, 6,and 8 is provided in the digital camera 2, the data processing unit 31according to this embodiment illustrated in FIG. 12 may be similarlyprovided in the digital camera 2 or may be provided in a computer 91 ora server 92 that is connected to the digital camera 2 through a wired orwireless network 94, as illustrated in FIG. 11. In addition, the dataprocessing unit 31 illustrated in FIG. 12 may be provided in, forexample, the computer 91 that can be connected to the main memory 10separated from the digital camera 2. Furthermore, the data processingunit 31 illustrated in FIG. 12, which will be described below, may beprovided in a portable terminal 96 such as a smart phone or a tabletterminal which is connected to the digital camera 2, the computer 91, orthe server 92 and can receive the original image data.

FIG. 12 is a block diagram illustrating an example of the functionalstructure of the data processing unit 31 (image processing unit 38)according to the fifth embodiment.

The data processing unit 31 according to this embodiment includes aprocessed data acquisition unit 60, a gain calculation unit 62, and a WBprocessing unit 64.

The processed data acquisition unit 60 acquires the original image data,the reference WB gains which are set for each light source type of theoriginal image data, and the mixture ratio of the reference WB gains setfor each light source type of the original image data from the mainmemory 10. In a case in which the reference WB gains and the mixtureratio are included in the tag information of the original image data,the processed data acquisition unit 60 can acquire the reference WBgains and the mixture ratio from the tag information of the acquiredoriginal image data.

The gain calculation unit 62 calculates the application WB gain for eachpixel of the original image data from the reference WB gains accordingto the mixture ratio.

The WB processing unit 64 applies the application WB gain to theoriginal image data to acquire WB-adjusted image data. For example, in acase in which the pixel value of the original image data is representedby “V_ij (where “i” indicates an X-axis value and “j” indicates a Y-axisvalue in a case in which a pixel position is represented by the X-axisvalue and the Y-axis value)” and the application WB gain is representedby “WB_ij (where “i” indicates an X-axis value and “j” indicates aY-axis value in a case in which a pixel position is represented by theX-axis value and the Y-axis value)“, each pixel value “Q_ij (where “i”indicates an X-axis value and “j” indicates a Y-axis value in a case inwhich a pixel position is represented by the X-axis value and the Y-axisvalue)” of the WB-adjusted image data is represented by the followingExpression 4.

Q_ij=V_ij×WB_ij   [Expression 4]

Next, a case in which the reference WB gains acquired by the processeddata acquisition unit 60 include a reference WB gain for flash lightwhich is set in a case in which the light source type is flash light anda reference WB gain for environmental light which is set in a case inwhich the light source type is environmental light will be described.The mixture ratio acquired by the processed data acquisition unit 60 isa mixture ratio of the reference WB gain for flash light and thereference WB gain for environmental light which are set for each pixelof the original image data. The gain calculation unit 62 calculates anapplication white balance gain for each pixel of the original image datafrom the reference WB gain for flash light and the reference WB gain forenvironmental light according to the mixture ratio.

According to the data processing unit 31, in a case in which theoriginal image data is RAW image data, when performing a developmentprocess for the RAW image data, the user can also perform multi-area WBprocessing, using the “reference WB gains (the reference WB gain forenvironmental light and the reference WB gain for flash light)” and the“mixture ratio” stored in the main memory 10.

FIG. 13 is a flowchart illustrating the flow of image processingaccording to the fifth embodiment.

First, the processed data acquisition unit 60 acquires the originalimage data, the reference WB gain WB1 for environmental light, thereference WB gain WB2 for flash light, and the mixture ratio Ratio_ijfrom the main memory 10 (S41 in FIG. 13).

Then, the gain calculation unit 62 acquires the application WB gainWB_ij for each pixel of the original image data from the reference WBgain WB1 for environmental light, the reference WB gain WB2 for flashlight, and the mixture ratio Ratio_ij (S42).

Then, the WB processing unit 64 applies the application WB gain WB_ij tothe original image data to acquire WB-adjusted image data (S43).

As described above, according to this embodiment, it is possible toappropriately perform multi-area WB processing for the original imagedata.

Since the reference WB gains (the reference WB gain for environmentallight and the reference WB gain for flash light) used in the multi-areaWB processing are prepared for each light source type, the user canperform WB processing in which the influence of some of the light sourcetypes is individually changed. For example, in a case in which the user“wants to slightly correct only an image portion (for example, a personimage) illuminated by flash to the red side, the WB processing unit 64may slightly shift the reference WB gain for flash light used in the WBprocessing to the red side. In this case, only the color (color balance)of an image portion, such as a person image, having a high proportion ofthe reference WB gain for flash light in the WB-adjusted image data canbe shifted to the red side and the correction has a small effect on abackground image portion having a low proportion of the reference WBgain for flash light and a high proportion of the reference WB gain forenvironmental light. As such, according to the image processing unit 38of this embodiment, flexibility in the adjustment of the white balanceis high and it is possible to simply obtain WB-adjusted image data thatmeets the demand of the user.

Sixth Embodiment

In this embodiment, the detailed description of the same or similarstructure and operation as those in the first to fifth embodiments willnot be repeated.

FIG. 14 is a block diagram illustrating an example of the functionalstructure of a data processing unit 31 (image processing unit 38)according to a sixth embodiment.

The data processing unit 31 according to this embodiment includes asetting mode acquisition unit 66 and a processing mode acquisition unit68, in addition to the processed data acquisition unit 60, the gaincalculation unit 62, and the WB processing unit 64.

The setting mode acquisition unit 66 acquires information about the WBsetting mode when the original image data is acquired. A method foracquiring the information about the WB setting mode is not particularlylimited. For example, in a case in which the information about the WBsetting mode is included in the tag information of the original imagedata, the setting mode acquisition unit 66 may read the informationabout the WB setting mode from the tag information of the original imagedata acquired by the processed data acquisition unit 60 and acquire theinformation about the WB setting mode.

The processing mode acquisition unit 68 acquires information about awhite balance processing mode (hereinafter, referred to as a “WBprocessing mode”) for the original image data. In this example, the “WBprocessing mode” is a WB mode that is desired by the user. The userinputs the WB mode that is desired as the actual WB processing for theoriginal image data as the “WB processing mode” to a processing modeinput unit 70.

The detailed form of the processing mode input unit 70 is notparticularly limited. In a case in which the processing mode input unit70 is provided in the digital camera 2, for example, the “operating unit9 (see FIG. 2)” may form the processing mode input unit 70. In addition,in a case in which the processing mode input unit 70 is provided in thecomputer 91, the server 92, or the portable terminal 96, the processingmode input unit 70 can be formed by an arbitrary operating unit, such asa keyboard, a mouse, or a touch panel which is provided in or connectedto these types of devices.

The “information about the WB processing mode for the original imagedata” acquired by the processing mode acquisition unit 68 is transmittedto the gain calculation unit 62. In addition, the “information about theWB setting mode for the original image data” acquired by the settingmode acquisition unit 66 is transmitted to the gain calculation unit 62.

In a case in which the WB processing mode is determined to be the sameas the WB setting mode on the basis of the information about the WBsetting mode and the information about the WB processing mode, the gaincalculation unit 62 calculates the application WB gain on the basis ofthe reference WB gain which is acquired from the main memory 10 throughthe processed data acquisition unit 60. On the other hand, in a case inwhich the WB processing mode is determined to be different from the WBsetting mode, the gain calculation unit 62 acquires the reference WBgain determined on the basis of the WB processing mode and calculatesthe application WB gain on the basis of the reference WB gain.

For example, in a case in which the WB setting mode is the AWB mode andthe WB processing mode is the preset WB mode, the gain calculation unit62 acquires the WB gain set for the preset WB mode and sets newreference WB gains (for example, the reference WB gain WB1 forenvironmental light and the reference WB gain WB2 for flash light) onthe basis of the reference WB gain for the preset WB mode. A method foracquiring the reference WB gain determined on the basis of the WBprocessing mode is not particularly limited. In a case in which the“reference WB gain determined on the basis of the WB processing mode” isstored as the tag information of the original image data in the mainmemory 10, the gain calculation unit 62 may read the stored informationand acquire the “reference WB gain determined on the basis of the WBprocessing mode”.

In a case in which the WB processing mode is determined to be differentfrom the WB setting mode, all of a plurality of reference WB gains whichare used by the gain calculation unit 62 to calculate the application WBgain may be changed or only some of the plurality of reference WB gainsmay be changed. For example, the user may input information about a WBgain that is desired to be changed from the WB setting mode and a WBgain after a detailed change among the reference WB gains (for example,the reference WB gain WB1 for environmental light and the reference WBgain WB2 for flash light) to the processing mode input unit 70. In thiscase, the processing mode acquisition unit 68 acquires the informationabout “the reference WB gain that is desired to be changed from the WBsetting mode” and “the WB gain after a detailed change” which are inputto the processing mode input unit 70 by the user and transmits theinformation to the gain calculation unit 62. The gain calculation unit62 changes “the reference WB gain that is desired to be changed from theWB setting mode” input by the user to the “WB gain after a detailedchange” and acquires the application WB gain on the basis of the changedreference WB gain and the mixture ratio.

FIG. 15 is a flowchart illustrating the flow of image processingaccording to the sixth embodiment. In the following example, a case inwhich the reference WB gain WB1 for environmental light and thereference WB gain WB2 for flash light are stored as the reference WBgains in the main memory 10 will be described.

First, the processed data acquisition unit 60 acquires the originalimage data, the reference WB gain WB1 for environmental light, thereference WB gain WB2 for flash light, and the mixture ratio Ratio_ijfrom the main memory 10 (S51 in FIG. 15).

Then, the setting mode acquisition unit 66 acquires information aboutthe WB setting mode when the original image data is acquired and theprocessing mode acquisition unit 68 acquires information about the WBprocessing mode which is input to the processing mode input unit 70 bythe user. Then, the gain calculation unit 62 determines whether the WBsetting mode is the same as the WB processing mode (S52).

In a case in which the WB setting mode is the same as the WB processingmode (Y in S52), the gain calculation unit 62 acquires the applicationWB gain WB_ij for each pixel of the original image data from thereference WB gains (the reference WB gain WB1 for environmental lightand the reference WB gain WB2 for flash light) and the mixture ratioRatio_ij which are acquired from the main memory 10 through theprocessed data acquisition unit 60 (S53).

On the other hand, in a case in which the WB setting mode is not thesame as the WB processing mode (N in S52), the gain calculation unit 62acquires the reference WB gain corresponding to the WB processing modeand sets the acquired reference WB gain as a new reference WB gain (thereference WB gain WB1 for environmental light and the reference WB gainWB2 for flash light) used in WB processing (S55). Then, the gaincalculation unit 62 acquires the application WB gain WB_ij set for eachpixel of the original image data from the newly set reference WB gainand the mixture ratio Ratio_ij (S53).

Then, the WB processing unit 64 applies the application WB gain WB_ij tothe original image data to acquire WB-adjusted image data (S54).

As described above, according to this embodiment, it is possible toperform multi-area WB processing in which the user's preference duringthe adjustment of the white balance is reflected.

Seventh Embodiment

In this embodiment, the detailed description of the same or similarstructure and operation as those in the first to sixth embodiments willnot be repeated.

This embodiment relates to an example in which some or all of thereference WB gains are replaced with the WB gain corresponding to thelight source type designated by the user and WB processing is performedfor original image data.

FIG. 16 is a block diagram illustrating an example of the functionalstructure of a data processing unit 31 (image processing unit 38)according to a seventh embodiment.

The data processing unit 31 according to this embodiment includes alight source type designation unit 72 and a designated gain acquisitionunit 74, in addition to the processed data acquisition unit 60, the gaincalculation unit 62, the WB processing unit 64, the setting modeacquisition unit 66, and the processing mode acquisition unit 68.

The light source type designation unit 72 receives information about thelight source type which is designated by the user through a light sourcetype input unit 76. The detailed form of the light source type inputunit 76 is not particularly limited. For example, similarly to theprocessing mode input unit 70, in a case in which the light source typeinput unit 76 is provided in the digital camera 2, the light source typeinput unit 76 can be formed by the “operating unit 9 (see FIG. 2)”. Inaddition, in a case in which the light source type input unit 76 isprovided in the computer 91, the server 92, or the portable terminal 96,the light source type input unit 76 can be formed by an arbitraryoperating unit, such as a keyboard, a mouse, or a touch panel which isprovided in or connected to these types of devices.

The designated gain acquisition unit 74 acquires the reference WB gainset for the “light source type designated by the user” which has beeninput to the light source type input unit 76 and then received by thelight source type designation unit 72.

The gain calculation unit 62 replaces at least some of the reference WBgains set for each light source type of the original image data with thereference WB gains acquired by the designated gain acquisition unit 74and calculates the application WB gain according to the mixture ratio.

Therefore, the user can replace some or all of the reference WB gains(for example, the reference WB gain WB1 for environmental light and thereference WB gain WB2 for flash light) with the reference WB gains setfor other types of light sources and can perform WB processing for theoriginal image data.

The processing mode input unit 70 and the light source type input unit76 may be integrally provided and the processing mode acquisition unit68 and the light source type designation unit 72 may be integrallyprovided. In this case, for example, in a case i which the user inputs alight source type for determining a new reference WB gain to the lightsource type input unit 76 in order to replace the reference WB gain, thegain calculation unit 62 may determine that the WB setting mode isdifferent from the WB processing mode.

FIG. 17 is a flowchart illustrating the flow of image processingaccording to the seventh embodiment. In the following description of theflowchart, for example, a case in which the reference WB gains of theoriginal image data are the reference WB gain WB1 for environmentallight and the reference WB gain WB2 for flash light will be described.

First, the processed data acquisition unit 60 acquires the originalimage data, the reference WB gain WB1 for environmental light, thereference WB gain WB2 for flash light, and the mixture ratio Ratio_ijfrom the main memory 10 (S61 in FIG. 17).

Then, the setting mode acquisition unit 66 acquires information aboutthe WB setting mode when the original image data is acquired and theprocessing mode acquisition unit 68 acquires information about the WBprocessing mode input by the user. Then, the gain calculation unit 62determines whether the WB setting mode is the same as the WB processingmode (S62).

In a case in which the WB setting mode is the same as the WB processingmode (Y in S62), the gain calculation unit 62 acquires the applicationWB gain WB_ij for each pixel of the original image data from thereference WB gains (the reference WB gain WB1 for environmental lightand the reference WB gain WB2 for flash light) and the mixture ratioRatio_ij which are acquired from the main memory 10 through theprocessed data acquisition unit 60 (S63).

On the other hand, in a case in which the WB setting mode is not thesame as the WB processing mode (N in S62), the user inputs informationabout light source types for determining the reference WB gains WB1 andWB2 to be newly set to the light source type input unit 76. The lightsource type designation unit 72 acquires data of the light source typesinput to the light source type input unit 76 and the designated gainacquisition unit 74 acquires new reference WB gains WB1 and WB2determined according to the input light source types (S65). Then, thegain calculation unit 62 acquires the application WB gain WB_ij set foreach pixel of the original image data from the new reference WB gainsWB1 and WB2 and the mixture ratio Ratio_ij (S63).

Then, the WB processing unit 64 applies the application WB gain WB_ij tothe original image data to acquire WB-adjusted image data (S64).

As described above, according to this embodiment, it is possible toperform multi-area WB processing in which the user's preference for thelight source type during the adjustment of the white balance isreflected.

In the above-described embodiment, an example in which, after it isdetermined whether the WB processing mode is different from the WBsetting mode, the user changes the light source type of the reference WBgain has been described. However, the user may change the light sourcetype of the reference WB gain, if necessary, without determining whetherthe WB processing mode is different from the WB setting mode. In thiscase, for example, in the example illustrated in FIG. 16, the processingmode input unit 70 and the processing mode acquisition unit 68 areunnecessary.

Eighth Embodiment

In this embodiment, the detailed description of the same or similarstructure and operation as those in the first to seventh embodimentswill not be repeated.

This embodiment relates to an aspect of the display of adjusted imagedata on the display unit 8 and white balance adjustment. Next, forexample, a case in which an image based on the WB-adjusted image datagenerated in the image processing unit 38 (data processing unit 31)according to the fifth embodiment (see FIG. 12) is displayed on thedisplay unit 8 will be described.

FIG. 18 is a block diagram illustrating an example of the functionalstructure of an image processing unit 38 according to an eighthembodiment.

The image processing unit 38 according to this embodiment includes adisplay control unit 35 and the display unit 8 which is controlled bythe display control unit 35, in addition to the data processing unit 31.The display control unit 35 is connected to the operating unit 9 throughthe system control unit 25 (see FIG. 2) and displays the image based onthe WB-adjusted image data on the display unit 8.

FIG. 19 is a diagram illustrating an example of the display of an imageon the display unit 8.

In the example illustrated in FIG. 19, the display control unit 35controls the display unit 8 such that the image based on adjusted imagedata is displayed in an image display portion 80 of the display unit 8.In addition, the display control unit 35 displays a light source displayportion 82 indicating the light source type of the original image dataon the display unit 8. In a case in which there are a plurality of lightsource types in the original image data, the display control unit 35displays a plurality of light source display portions 82 provided foreach light source type on the display unit 8. Therefore, for example, ina case in which environmental light and flash light are assumed as thelight sources of the original image data, a light source display portion82 a (“WB1 setting” in the example illustrated in FIG. 19) related tothe environmental light and a light source display portion 82 b (“WB2setting” in the example illustrated in FIG. 19) related to the flashlight are displayed on the display unit 8.

In the example illustrated in FIG. 19, adjusted image data is generatedusing flash emission image data as the original image data, and an imagebased on the adjusted image data includes a person image and thebackground including a building. Therefore, in the image based on theadjusted image data, a person image portion is greatly affected by flashlight and a back ground portion is greatly affected by environmentallight.

In this embodiment, the user designates any one of the light sources ofthe original image data and an image portion (pixel) that is greatlyaffected by the designated light source is displayed in the imagedisplay portion 80 so as to be highlighted.

FIG. 20 is a diagram illustrating an example of the display of an imageon the display unit 8 in the mode in which the user designateshighlighting. FIG. 21 is a diagram illustrating an example of thedisplay of an image on the display unit 8 in a case in which the userdesignates highlighting.

In a case in which a plurality of light source display portions 82 aredisplayed on the display unit 8, the display control unit 35 displaysthe plurality of light source display portions 82 such that any one ofthe plurality of light source display portions 82 can be designated by ahighlighting designation portion 84 in response to an operation of theuser through the operating unit 9. In addition, the display control unit35 sets a portion of or the entire image displayed in the image displayportion 80 of the display unit 8 as a highlighted portion 86 accordingto the influence rate of the light source corresponding to the lightsource display portion 82 designated by the highlighting designationportion 84 through the operating unit 9. Since the influence rate of thelight source can be determined on the basis of the mixture ratio, thedisplay control unit 35 acquires the influence rate of the light sourcefor each pixel of the image displayed in the image display portion 80,on the basis of the mixture ratio acquired from the main memory 10through the processed data acquisition unit 60.

Specifically, the display control unit 35 may display a pixel, which ismore affected by a light source corresponding to the light sourcedisplay portion 82 that is designated by the highlighting designationportion 84 in response to an operation of the user through the operatingunit 9 than by other types of light sources among the light sources ofthe original image data, among the pixels of the image based on theadjusted image data displayed in the image display portion 80 as thehighlighted portion 86 in the image display portion 80 of the displayunit 8. For example, in a case in which the light source types of theoriginal image data are environmental light (corresponding to the “lightsource display portion 82 a” illustrated in FIG. 21) and flash light(corresponding to the “light source display portion 82 b”), when theuser operates the operating unit 9 to designate the light source displayportion 82 b corresponding to the flash light with the highlightingdesignation portion 84 as illustrated in FIG. 20, the pixels of a personimage portion which is more affected by flash light than byenvironmental light are to be displayed as the highlighted portion 86 inthe image display portion 80 as illustrated in FIG. 21.

A criterion for determining the highlighted portion 86 is notparticularly limited. The display control unit 35 may determine thepixels which will be the highlighted portion 86 according to theabsolute value of the influence rate of the light source designated bythe user, instead of comparing the influence rates of the light sourcesof the original image data. That is, the display control unit 35 maydisplay a pixel, on which the influence rate of the light sourcecorresponding to the light source display portion 82 designated by theuser through the operating unit 9 is higher than a first rate, in theimage based on the adjusted image data displayed in the image displayportion 80 on the display unit 8 so as to be highlighted. The “firstrate” can be set to any value. For example, the first rate can be set toa percentage of 50%.

A method of displaying the highlighted portion 86 is not particularlylimited. For example, the highlighted portion 86 may be displayed insuch a way that an area formed by the pixels to be highlighted issurrounded by a thick frame.

As described above, according to this embodiment, when the userdesignates the light source display portion 82, an image portion (pixel)on which the influence rate of a specific light source is high isdisplayed as the highlighted portion 86 in the image display portion 80.Therefore, the user can easily check a portion on which the influencerate of a specific light source type is high in the image displayed inthe image display portion 80.

Ninth Embodiment

In this embodiment, the detailed description of the same or similarstructure and operation as those in the eighth embodiments will not berepeated.

In this embodiment, the light source type designated by the user can bechanged or the influence rate can be changed. Next, an example in whichthe light source type designated by the user can be changed will bedescribed.

FIG. 22 is a block diagram illustrating an example of the functionalstructure of an image processing unit 38 according to a ninthembodiment.

The image processing unit 38 according to this embodiment includes awhite balance adjustment unit (hereinafter, referred to as a “WBadjustment unit”) 90 that adjusts the white balance of an image, inaddition to the data processing unit 31, the display control unit 35,and the display unit 8.

The other structures of the image processing unit 38 are the same asthose of the image processing unit 38 (see FIG. 18) according to theeighth embodiment.

That is, the display control unit 35 is connected to the operating unit9 operated by the user through the system control unit 25 (see FIG. 3).As illustrated in FIG. 19, the display control unit 35 displays theimage based on adjusted image data in the image display portion 80 ofthe display unit 8 and displays the light source display portion 82indicating the light source type of the original image data on thedisplay unit 8. In a case in which there are a plurality of light sourcetypes in the original image data, the display control unit 35 displays aplurality of light source display portions 82 provided for each lightsource type on the display unit 8.

Then, in a case in which a plurality of light source display portions 82are displayed on the display unit 8, the display control unit 35displays the plurality of light source display portions 82 such that anyone of the plurality of light source display portions 82 can bedesignated by the highlighting designation portion 84 (see FIG. 20) inresponse to an operation of the user through the operating unit 9. Whenany one of the plurality of light source display portions 82 isdesignated by the highlighting designation portion 84, the displaycontrol unit 35 displays a change display portion 88 on the display unit8 as illustrated in FIG. 23. The change display portion 88 receives achange in the white balance based on the light source corresponding tothe light source display portion 82 which is designated by thehighlighting designation portion 84 through the operating unit 9.

The display control unit 35 receives a change in the white balance of animage through the change display portion 88 in response to an operationof the user through the operating unit 9. In the example illustrated inFIG. 23, the user can designate a light source that is used, instead ofthe light source (flash light) corresponding to the light source displayportion 82 b in which “WB2 setting” is displayed from other lightsources (for example, “daylight”, “shade”, a “fluorescent light”, and a“light bulb”) displayed in the change display portion 88 through theoperating unit 9.

When the display control unit 35 receives a change in the white balance,the WB adjustment unit 90 changes and adjusts the white balance of theimage based on the adjusted image data displayed in the image displayportion 80 and generates image data (hereinafter, referred to as“changed image data”) in which a change in the light source to a newlight source designated by the user is reflected.

Then, the display control unit 35 displays an image based on the changedimage data of which the white balance has been changed and adjusted bythe WB adjustment unit 90 in the image display portion 80 of the displayunit 8.

Instead of the light source, the influence rate of the light sourcedesignated by the user may be changed. That is, in a case in which aplurality of light source display portions 82 are displayed on thedisplay unit 8, when any one of the plurality of light source displayportions 82 is designated by the user through the operating unit 9, thedisplay control unit 35 may display a change display portion 88, whichreceives a change in the influence rate of the light sourcecorresponding to the light source display portion 82 designated throughthe operating unit 9, in the image display portion. Then, the displaycontrol unit 35 may receive the amount of change in the “influence rateof the light source corresponding to the designated light source display82” in the image through the change display portion in response to anoperation of the user through the operating unit 9. In the exampleillustrated in FIG. 23, the display control unit 35 may display apercentage in the range of, for example, “0%” to “100%” on the displayunit 8 in any aspect, instead of displaying “daylight”, “shade”, a“fluorescent light”, and a “light bulb” in the change display portion88. In this case, the user can select any percentage displayed in thechange display portion 88 through the operating unit 9. The percentageselected by the user may be used as the “amount of change in theinfluence rate of the light source corresponding to the designated lightsource display portion 82.

In this case, the WB adjustment unit 90 (see FIG. 22) adjusts the whitebalance of the image based on the adjusted image data, reflects theamount of change in the influence rate received by the display controlunit 35 through the light source display portion 82 in the image, andgenerate changed image data. Then, the display control unit 35 displaysan image based on the changed image data of which the white balance hasbeen changed and adjusted by the WB adjustment unit 90 in the imagedisplay portion 80 of the display unit 8.

As described above, according to this embodiment, the user can changethe white balance of the image for each light source. In particular, thedisplay unit 8 (image display portion 80) displays the pixel on whichthe influence rate of the light source designated as a change target ishigh so as to be highlighted. Therefore, the user can easily determine apart of the image of which the color (color balance) is to be changed.

MODIFICATION EXAMPLES

Among the above-described embodiments and modification examples, anyembodiments and modification examples may be combined with each other.The above-described embodiments are illustrative and the invention maybe applied to other structures. Each of the above-mentioned functionalstructures can be implemented by any hardware, software, or acombination thereof. For example, the invention can also be applied to aprogram that causes a computer to perform an image processing method(image processing procedure) in each of the above-mentioned devices andprocessing units (for example, the data processing unit 31), acomputer-readable storage medium (non-transitory storage medium) thatstores the program, or a computer in which the program can be installed.

The aspects to which the invention can be applied are not limited to thedigital camera and the computer (server). The invention can also beapplied to cameras having an imaging function as a main function andmobile devices having functions (a calling function, a communicationfunction, and other computer functions) other than the imaging functionin addition to the imaging function. Other aspects to which theinvention can be applied are, for example, mobile phones, smart phones,personal digital assistants (PDAs), and portable game machines with acamera function. Hereinafter, an example of the smart phone to which theinvention can be applied will be described.

<Structure of Smart Phone>

FIG. 24 is a diagram illustrating the outward appearance of a smartphone 101. The smart phone 101 illustrated in FIG. 24 comprises ahousing 102 with a flat panel shape and a display input unit 120 havinga display panel 121 as a display unit and an operation panel 122 as aninput unit which are integrally formed on one surface of the housing102. The housing 102 comprises a speaker 131, a microphone 132, anoperating unit 140, and a camera unit 141. However, the configuration ofthe housing 102 is not limited thereto. For example, the display unitand the input unit may be independently provided, or the housing 102 mayhave a folding structure or a sliding structure.

FIG. 25 is a block diagram illustrating an example of the structure ofthe smart phone 101 illustrated in FIG. 24. As illustrated in FIG. 25,the smart phone 101 comprises, as main components, a wirelesscommunication unit 110, the display input unit 120, a calling unit 130,the operating unit 140, the camera unit 141, a storage unit 150, anexternal input/output unit 160, a global positioning system (GPS)receiving unit 170, a motion sensor unit 180, a power supply unit 190,and a main control unit 100. The smart phone 101 has, as a mainfunction, a wireless communication function which performs mobilewireless communication through a base station apparatus and a mobilecommunication network.

The wireless communication unit 110 performs wireless communication withthe base station apparatus which is accommodated in the mobilecommunication network in response to an instruction from the maincontrol unit 100. The wireless communication is used to transmit andreceive various types of file data, such as voice data and image data,and electronic mail data or to receive, for example, web data andstreaming data.

The display input unit 120 is a so-called touch panel that displays, forexample, images (still images and moving images) or text information tovisually transmit information to the user and detects the user'soperation for the displayed information under the control of the maincontrol unit 100 and comprises the display panel 121 and the operationpanel 122.

The display panel 121 uses, for example, a liquid crystal display (LCD)or an organic electro-luminescence display (OELD) as a display device.The operation panel 122 is a device that is provided such that an imagedisplayed on a display surface of the display panel 121 is visuallyrecognized and detects coordinates operated by a finger of the user or astylus. When the device is operated by a finger of the user or a stylus,a detection signal which is generated by the operation is output to themain control unit 100. Then, the main control unit 100 detects anoperation position (coordinates) on the display panel 121 on the basisof the received detection signal.

As illustrated in FIG. 24, the display panel 121 and the operation panel122 of the smart phone 101 which is an embodiment of the imaging deviceaccording to the invention are integrated to form the display input unit120 and the operation panel 122 is arranged so as to completely coverthe display panel 121. In a case in which this arrangement is used, theoperation panel 122 may have a function of detecting the user'soperation even in a region other than the display panel 121. In otherwords, the operation panel 122 may comprise a detection region(hereinafter, referred to as a display region) for an overlap portionwhich overlaps the display panel 121 and a detection region(hereinafter, referred to as a non-display region) for an outer edgeportion which does not overlap the display panel 121.

The size of the display region may be exactly equal to the size of thedisplay panel 121. However, the sizes are not necessarily equal to eachother. The operation panel 122 may comprise two sensitive regions, thatis, an outer edge portion and an inner portion other than the outer edgeportion. The width of the outer edge portion is appropriately designedaccording to, for example, the size of the housing 102. Examples of aposition detecting method which is used in the operation panel 122include a matrix switching method, a resistive film method, a surfaceelastic wave method, an infrared method, an electromagnetic inductionmethod, and a capacitive sensing method. Any of the methods may be used.

The calling unit 130 comprises the speaker 131 and the microphone 132.The calling unit 130 converts the voice of the user which is inputthrough the microphone 132 into voice data which can be processed by themain control unit 100 and outputs the converted voice data to the maincontrol unit 100. In addition, the calling unit 130 decodes voice datareceived by the wireless communication unit 110 or the externalinput/output unit 160 and outputs the decoded voice data from thespeaker 131. As illustrated in FIG. 24, for example, the speaker 131 canbe mounted on the same surface as the display input unit 120 and themicrophone 132 can be mounted on a side surface of the housing 102.

The operating unit 140 is a hardware key which uses, for example, a keyswitch and receives instructions from the user. For example, asillustrated in FIG. 24, the operating unit 140 is a push button switchwhich is mounted on the side surface of the housing 102 of the smartphone 101, is turned on when it is pressed by, for example, a finger,and is turned off by the restoring force of a spring when the finger istaken off.

The storage unit 150 stores a control program or control data of themain control unit 100, application software, address data which isassociated with, for example, the names or phone numbers ofcommunication partners, transmitted and received electronic mail data,web data which is downloaded by web browsing, and downloaded contentdata. In addition, the storage unit 150 temporarily stores, for example,streaming data. The storage unit 150 includes an internal storage unit151 which is provided in the smart phone and an external storage unit152 which has a detachable external memory slot. The internal storageunit 151 and the external storage unit 152 forming the storage unit 150are implemented by a storage medium, such as a flash memory, a harddisk, a multimedia-card-micro-type memory, a card-type memory (forexample, a MicroSD (registered trademark) memory), a random accessmemory (RAM), or a read only memory (ROM).

The external input/output unit 160 functions as an interface with all ofthe external apparatuses connected to the smart phone 101 and isdirectly or indirectly connected to other external apparatuses bycommunication (for example, universal serial bus (USB) communication orIEEE1394 defined by the Institute of Electrical and ElectronicsEngineers, Inc. (IEEE)) or a network (for example, the Internet, awireless local area network (LAN), a Bluetooth (registered trademark)network, a radio frequency identification (RFID) network, an infrareddata association (IrDA) (registered trademark) network, an ultrawideband (UWB) (registered trademark) network, or a ZigBee (registeredtrademark) network).

Examples of the external apparatus connected to the smart phone 101include a wired/wireless headset, a wired/wireless external charger, awired/wireless data port, a memory card or a subscriber identity module(SIM) card/user identity module (UIM) card which is connected through acard socket, an external audio/video apparatus which is connectedthrough audio/video input/output (I/O) terminals, a wirelessly connectedexternal audio/video apparatus, a smart phone which is connectedwirelessly or in a wired manner, a personal computer which is connectedwirelessly or in a wired manner, a PDA which is connected wirelessly orin a wired manner, and an earphone which is connected wirelessly or in awired manner. The external input/output unit may transmit data which isreceived from the external apparatus to each component of the smartphone 101 or may transmit data in the smart phone 101 to the externalapparatus.

The GPS receiving unit 170 receives GPS signals transmitted from GPSsatellites ST1 to STn and performs a position measurement process on thebasis of the received GPS signals to detect a position including thelatitude, longitude, and height of the smart phone 101, in response toan instruction from the main control unit 100. In a case in which theGPS receiving unit 170 can acquire positional information from thewireless communication unit 110 or the external input/output unit 160(for example, the wireless LAN), the GPS receiving unit 170 can detectthe position using the positional information.

The motion sensor unit 180 comprises, for example, a triaxialacceleration sensor and detects the physical movement of the smart phone101 in response to an instruction from the main control unit 100. Whenthe physical movement of the smart phone 101 is detected, the movingdirection or acceleration of the smart phone 101 is detected. Thedetection result is output to the main control unit 100.

The power supply unit 190 supplies power which is stored in a battery(not illustrated) to each unit of the smart phone 101 in response to aninstruction from the main control unit 100.

The main control unit 100 comprises a microprocessor, operates on thebasis of the control program or control data stored in the storage unit150, and controls the overall operation of each unit of the smart phone101. The main control unit 100 has an application processing functionand a mobile communication control function of controlling each unit ofa communication system in order to perform voice communication or datacommunication through the wireless communication unit 110.

The application processing function is implemented by the operation ofthe main control unit 100 based on the application software which isstored in the storage unit 150. Examples of the application processingfunction include an infrared communication function which controls theexternal input/output unit 160 such that data communication with anopposing apparatus is performed, an electronic mail function whichtransmits and receives electronic mail, and a web browsing functionwhich browses web pages.

The main control unit 100 has, for example, an image processing functionwhich displays an image on the display input unit 120 on the basis ofimage data (still image or moving image data) such as received data ordownloaded streaming data. The image processing function means thefunction of the main control unit 100 decoding the image data,performing image processing on the decoding result, and displaying theimage on the display input unit 120.

The main control unit 100 performs display control for the display panel121 and operation detection control for detecting the operation of theuser through the operating unit 140 and the operation panel 122.

The main control unit 100 performs the display control to display asoftware key, such as an icon for starting application software or ascroll bar, or to display a window for writing electronic mail. Thescroll bar means a software key for receiving an instruction to move adisplayed portion of an image that is too large to fit into the displayregion of the display panel 121.

The main control unit 100 performs the operation detection control todetect the operation of the user input through the operating unit 140,to receive an operation for the icon or the input of a character stringto an input field of the window through the operation panel 122, or toreceive a request to scroll the displayed image through the scroll bar.

In addition, the main control unit 100 has a touch panel controlfunction that performs the operation detection control to determinewhether the position of an operation for the operation panel 122 is anoverlap portion (display region) which overlaps the display panel 121 oran outer edge portion (non-display region) which does not overlap thedisplay panel 121 other than the overlap portion and controls asensitive region of the operation panel 122 or the display position ofthe software key.

The main control unit 100 can detect a gesture operation for theoperation panel 122 and can perform a predetermined function accordingto the detected gesture operation. The gesture operation does not mean asimple touch operation according to the related art, but means anoperation which draws a trace using a finger, an operation whichdesignates a plurality of positions at the same time, or a combinationthereof which draws a trace for at least one of the plurality ofpositions.

The camera unit 141 is a digital camera which captures an image using animaging element such as a CMOS. In addition, the camera unit 141 canconvert captured image data into image data which is compressed in, forexample, a JPEG format, stores the converted image data in the storageunit 150, and outputs the converted image data through the externalinput/output unit 160 or the wireless communication unit 110, under thecontrol of the main control unit 100. As illustrated in FIG. 24, thecamera unit 141 is mounted on the same surface as the display input unit120 in the smart phone 101. However, the mounting position of the cameraunit 141 is not limited thereto. For example, the camera unit 141 may bemounted on the rear surface of the display input unit 120 or a pluralityof camera units 141 may be mounted. In a case in which a plurality ofcamera units 141 are mounted, the camera units 141 which are used tocapture images may be switched such that a single camera unit capturesimages or the plurality of camera units 141 may be simultaneously usedto capture images.

The camera unit 141 can be used for various functions of the smart phone101. For example, the image captured by the camera unit 141 can bedisplayed on the display panel 121 or the image captured by the cameraunit 141 can be used as one of the operation inputs of the operationpanel 122. When the GPS receiving unit 170 detects the position, theposition may be detected with reference to the image from the cameraunit 141. In addition, the optical axis direction of the camera unit 141in the smart phone 101 may be determined or the current usageenvironment may be determined, with reference to the image from thecamera unit 141, using the triaxial acceleration sensor or without usingthe triaxial acceleration sensor. Of course, the image from the cameraunit 141 may be used in the application software.

For example, the positional information which is acquired by the GPSreceiving unit 170, the voice information which is acquired by themicrophone 132 (for example, the main control unit may convert the voiceinformation into text information), and the posture information which isacquired by the motion sensor unit 180 may be added to the image data ofa still image or a moving image and the image data may be stored in thestorage unit 150 and may be output through the external input/outputunit 160 or the wireless communication unit 110.

The image processing unit 38 (see FIGS. 4, 6, 8, 12, 14, 16, 18, and 22)may be implemented by, for example, the main control unit 100.

EXPLANATION OF REFERENCES

2: digital camera

3: camera body

4: lens barrel

5: flash light emitting unit

6: shutter button

7: power switch

8: display unit

9: operating unit

10: main memory

12: lens unit

20: mechanical shutter

21: imaging element

22: process processing unit

23: A/D conversion unit

24: buffer memory

25: system control unit

26: shutter driving unit

27: lens driving unit

28: power supply control unit

29: power supply

30: control memory

31: data processing unit

32: compression and decompression unit

33: storage control unit

34: clock device

35: display control unit

36: user interface

38: image processing unit

40: light source type determination unit

42: gain acquisition unit

44: mixture ratio acquisition unit

46: flash image acquisition unit

48: setting mode determination unit

60: processed data acquisition unit

62: gain calculation unit

64: WB processing unit

66: setting mode acquisition unit

68: processing mode acquisition unit

70: processing mode input unit

72: light source type designation unit

74: designated gain acquisition unit

76: light source type input unit

80: image display portion

82: light source display

84: highlighting designation portion

86: highlighted portion

88: change display

90: WB adjustment unit

91: computer

92: server

94: network

96: portable terminal

100: main control unit

101: smart phone

102: housing

110: wireless communication unit

120: display input unit

121: display panel

122: operation panel

130: calling unit

131: speaker

132: microphone

140: operating unit

141: camera unit

150: storage unit

151: internal storage unit

152: external storage unit

160: external input/output unit

170: GPS receiving unit

180: motion sensor unit

190: power supply unit

What is claimed is:
 1. An image processing device comprising: aprocessed data acquisition unit that acquires original image data, areference white balance gain which is set for each light source type ofthe original image data, and a mixture ratio of the reference whitebalance gains set for each pixel of the original image data from astorage medium; and a gain calculation unit that calculates anapplication white balance gain for each pixel of the original image datafrom the reference white balance gains according to the mixture ratio.2. The image processing device according to claim 1, wherein thereference white balance gains acquired by the processed data acquisitionunit include a reference white balance gain for flash light which is setin a case in which the light source type is flash light and a referencewhite balance gain for environmental light which is set in a case inwhich the light source type is environmental light, the mixture ratioacquired by the processed data acquisition unit is a mixture ratio ofthe reference white balance gain for flash light and the reference whitebalance gain for environmental light which are set for each pixel of theoriginal image data, and the gain calculation unit calculates theapplication white balance gain for each pixel of the original image datafrom the reference white balance gain for flash light and the referencewhite balance gain for environmental light according to the mixtureratio.
 3. The image processing device according to claim 1, furthercomprising: a setting mode acquisition unit that acquires informationabout a white balance setting mode when the original image data isacquired; and a processing mode acquisition unit that acquiresinformation about a white balance processing mode for the original imagedata, wherein the gain calculation unit calculates the application whitebalance gain on the basis of the reference white balance gain acquiredfrom the storage medium in a case in which the white balance processingmode is determined to be the same as the white balance setting mode onthe basis of the information about the white balance setting mode andthe information about the white balance processing mode, acquires thereference white balance gain which is set on the basis of the whitebalance processing mode in a case in which the white balance processingmode is determined to be different from the white balance setting mode,and calculates the application white balance gain on the basis of thereference white balance gain.
 4. The image processing device accordingto claim 1, further comprising: a light source type designation unitthat receives a light source type designated by a user; and a designatedgain acquisition unit that acquires a reference white balance gain setfor the light source type designated by the user which is received bythe light source type designation unit, wherein the gain calculationunit replaces at least some of the reference white balance gains set foreach light source type of the original image data with the referencewhite balance gain acquired by the designated gain acquisition unit andcalculates the application white balance gain according to the mixtureratio.
 5. The image processing device according to claim 1, furthercomprising: a white balance processing unit that applies the applicationwhite balance gain to the original image data to acquirewhite-balance-adjusted image data.
 6. The image processing deviceaccording to claim 5, further comprising: a display control unit; and adisplay unit that is controlled by the display control unit, wherein thedisplay control unit displays an image based on thewhite-balance-adjusted image data on the display unit.
 7. The imageprocessing device according to claim 6, wherein the display control unitis connected to an operating unit that is operated by the user, thedisplay control unit displays a light source display portion indicatingthe light source type of the original image data on the display unit anddisplays a plurality of light source display portions which are providedfor each light source type on the display unit in a case in which thereare a plurality of light source types in the original image data, and ina case in which the plurality of light source display portions aredisplayed on the display unit, when the user operates the operating unitto designate any one of the plurality of light source display portions,the display control unit displays a portion of or the entire imagedisplayed on the display unit so as to be highlighted according to aninfluence rate of a light source corresponding to the light sourcedisplay portion designated through the operating unit.
 8. The imageprocessing device according to claim 7, wherein the display control unitdisplays a pixel, which is more affected by the light sourcecorresponding to the light source display portion that is designated bythe user through the operating unit than by other types of light sourcesamong the light sources of the original image data, among pixels of theimage on the display unit so as to be highlighted.
 9. The imageprocessing device according to claim 7, wherein the display control unitdisplays a pixel, on which the influence rate of the light sourcecorresponding to the light source display portion that is designated bythe user through the operating unit is higher than a first rate, amongthe pixels of the image on the display unit so as to be highlighted. 10.The image processing device according to claim 6, further comprising: awhite balance adjustment unit that adjusts a white balance of the image,wherein the display control unit is connected to an operating unit thatis operated by the user, the display control unit displays a lightsource display portion indicating the light source type of the originalimage data on the display unit and displays a plurality of light sourcedisplay portions which are provided for each light source type on thedisplay unit in a case in which there are a plurality of light sourcetypes in the original image data, in a case in which the plurality oflight source display portions are displayed on the display unit, whenthe user operates the operating unit to designate any one of theplurality of light source display portions, the display control unitdisplays a change display portion for receiving a change in the whitebalance based on the light source corresponding to the light sourcedisplay portion which is designated through the operating unit on thedisplay unit, the display control unit receives the change in the whitebalance of the image through the change display portion in response toan operation of the user through the operating unit, when the displaycontrol unit receives the change in the white balance, the white balanceadjustment unit adjusts the white balance of the image and reflects thechange in the image, and the display control unit displays the image ofwhich the white balance has been adjusted by the white balanceadjustment unit on the display unit.
 11. The image processing deviceaccording to claim 6, further comprising: a white balance adjustmentunit that adjusts a white balance of the image, wherein the displaycontrol unit is connected to an operating unit that is operated by theuser, the display control unit displays a light source display portionindicating the light source type of the original image data on thedisplay unit and displays a plurality of light source display portionswhich are provided for each light source type on the display unit in acase in which there are a plurality of light source types in theoriginal image data, in a case in which the plurality of light sourcedisplay portions are displayed on the display unit, when the useroperates the operating unit to designate any one of the plurality oflight source display portions, the display control unit displays achange display portion for receiving a change in an influence rate ofthe light source corresponding to the light source display portion whichis designated through the operating unit in the image on the displayunit, the display control unit receives an amount of change in theinfluence rate in the image through the change display portion inresponse to an operation of the user through the operating unit, thewhite balance adjustment unit adjusts the white balance of the image andreflects the amount of change in the influence rate which has beenreceived by the display control unit through the change display portionin the image, and the display control unit displays the image of whichthe white balance has been adjusted by the white balance adjustment uniton the display unit.